PaperlayerView exact revision on arXiv

Source-linked AI summary

RECALL: Replay-based Continual Learning in Semantic Segmentation

Andrea Maracani, Umberto Michieli, Marco Toldo, Pietro Zanuttigh

arXiv:2108.03673v2cs.CV

TL;DR

Continual semantic segmentation must learn new classes without access to prior training data, while existing approaches struggle with forgetting across multiple steps and background shift. RECALL reconstructs old-class data through GAN-based or web-crawled replay and uses background inpainting during incremental training. The framework improves accuracy and robustness across multiple incremental-step scenarios compared with competing methods.

  • Problem

    Continual semantic segmentation must learn new classes incrementally without prior training data, while catastrophic forgetting, multiple steps, and background shift remain challenging.

  • Method

    RECALL re-creates old-class data through conditional GAN sampling or web crawling, blends replay with new data, and applies background inpainting to current samples.

  • Results

    RECALL improves accuracy and robustness across multiple incremental-step scenarios compared with competing methods, achieving state-of-the-art results across a wide range of scenarios.

  • Takeaways & Limitations

    Replay and background inpainting provide a framework for continual semantic segmentation that does not store samples from past training stages.

  • Takeaways & Limitations

    The web-retrieval strategy requires internet access and provides limited control over weak labels from the web source.

Abstract

from arXiv · show

Deep networks allow to obtain outstanding results in semantic segmentation, however they need to be trained in a single shot with a large amount of data. Continual learning settings where new classes are learned in incremental steps and previous training data is no longer available are challenging due to the catastrophic forgetting phenomenon. Existing approaches typically fail when several incremental steps are performed or in presence of a distribution shift of the background class. We tackle these issues by recreating no longer available data for the old classes and outlining a content inpainting scheme on the background class. We propose two sources for replay data. The first resorts to a generative adversarial network to sample from the class space of past learning steps. The second relies on web-crawled data to retrieve images containing examples of old classes from online databases. In both scenarios no samples of past steps are stored, thus avoiding privacy concerns. Replay data are then blended with new samples during the incremental steps. Our approach, RECALL, outperforms state-of-the-art methods.

1. Introduction

Continual semantic segmentation must learn new classes over multiple steps while preserving earlier knowledge, but existing methods struggle with catastrophic forgetting, multiple steps, and background shift. RECALL addresses these challenges by replaying reconstructed old-class data and inpainting background regions during incremental training.

  • Catastrophic forgetting causes knowledge of previously seen classes to be lost when models train on novel classes without additional provisions.
  • Existing class-incremental segmentation methods often fail after multiple incremental steps or when background statistics shift across learning steps.Background shift occurs because the background can incorporate old or future classes.
  • RECALL replaces teacher-model distillation with replay strategies that recreate old-class representations and mix them with data containing newly learned classes.
  • RECALL combines replay with background inpainting to address forgetting and background shift in continual semantic segmentation.The paper presents these components as its main contributions and reports state-of-the-art results across multiple scenarios.
  • Replay data come from either a class-conditioned GAN trained on another dataset or web-crawled images retrieved through class-name queries.Both sources provide weakly labeled data, which are converted into segmentation pseudo-labels by a side labeling module.

3. Problem Formulation

RECALL addresses continual semantic segmentation, where only new-class data remain available and background semantics shift across steps. Its replay block retrieves or generates old-class images, labels them, inpaints background regions, and trains on replay-augmented data.

  • Problem Formulation: Continual semantic segmentation adds disjoint class sets across incremental steps while prior training data are unavailable.The model expands its learnable class collection at each step, with only the current partition accessible.
  • Problem Formulation: Background undergoes semantic shift because pixels from changing class sets are repeatedly assigned to the background category.
  • General Architecture: RECALL uses a Replay Block to retrieve old-class images and obtain reliable semantic labels from learned past knowledge.The Source Block supplies images, while separate label-evaluation modules annotate them.
  • General Architecture: The Source Block uses either a web-based source or a GAN to provide images associated with past semantic classes.
  • General Architecture: Background self-inpainting replaces current background labels with predictions from the previous model at each incremental step.Step zero is not inpainted because no prior model knowledge is available.
  • General Architecture: Replay images and current samples form an augmented training partition containing old and new classes, reducing bias toward new classes and preventing forgetting.
  • General Architecture: The framework trains the main segmentation model with replay-augmented data while separately training helper decoders and source components for future replay.

5. Replay Strategies

RECALL supports replay through a conditional GAN or web crawling, maps target classes when needed, and labels retrieved images before incremental training. The web alternative is simpler but depends on internet access and uncontrolled weak labels.

  • Replay by GAN: The GAN strategy generates replay images from a conditional GAN pre-trained on a broad visual dataset.A class-mapping module translates segmentation classes into the GAN label space before generation.
  • Replay by GAN: GAN replay requires statistically similar mapped classes for generated samples to resemble the target segmentation classes.
  • Replay by GAN: At each incremental step, an image classifier identifies the GAN class with the highest aggregate probability for each current class.
  • Replay by GAN: Past classes are mapped to stored GAN correspondences, generated images are produced, and the Label Evaluation Block assigns semantic labels.
  • Replay by Web Crawler: The web strategy downloads images whose tags and descriptions contain each past class name, then sends them to the Label Evaluation Block.
  • Replay by Web Crawler: Web retrieval introduces no learnable modules and avoids assuming a large, diverse pre-training dataset, but requires internet access and offers limited control over weak labels.

6. Implementation Details

The implementation uses DeepLab-V2 with a ResNet-101 backbone, freezes the encoder during incremental steps, and trains with polynomially decayed SGD. GAN replay uses BigGAN-deep with 500 samples per old class.

  • Architecture and Optimization: RECALL uses DeepLab-V2 with a ResNet-101 backbone and is designed to remain independent of the specific network architecture.
  • Architecture and Optimization: During incremental steps, only the main decoder and helper decoders are trained while the encoder remains fixed.
  • Architecture and Optimization: SGD with momentum uses an initial learning rate of 5 × 10^-4, polynomial decay with power 0.9, and setup-dependent training lengths.Training lasts |Ck|×1000 steps in the disjoint setup and |Ck|×1500 steps in the overlapped setup.
  • Replay Configuration: The helper decoder uses a learning rate decaying from 2 × 10^-4 to 2 × 10^-6, while BigGAN-deep generates 500 replay samples per old class.The interleaving ratio rold/rnew is set to 1.
  • Runtime and Pre-processing: Training uses random scaling, mirroring, padding, and cropping to 321 × 321 pixels on a single NVIDIA RTX 2070 Super.The longest reported training run takes about 5 hours.

7. Experimental Results

RECALL is evaluated on Pascal VOC 2012 across six class-incremental setups, including increasingly difficult multi-step scenarios. Replay and background inpainting improve robustness, with the strongest gains appearing as incremental complexity increases.

  • Main Results: RECALL achieves higher overall mIoU than competitors in 19-1 while retaining accuracy on previously learned classes.The authors attribute this to the combination of replay strategies and background inpainting.
  • Main Results: RECALL maintains mIoU of 61.9%, 60.6%, and 57.8% across 10-10, 10-5, and 10-1, while FT and SDR decline more sharply.The 10-1 per-step evolution also shows higher mIoU than competitors throughout learning.
  • Qualitative Analysis: Qualitative results show RECALL preserving old classes such as bus, cow, and horse while reducing confusion with newly introduced classes.Competing methods more often replace or mix old-class predictions with novel classes.
  • Memory Analysis: Web replay has memory occupation close to standard methods, whereas storing all training images grows rapidly; GAN storage remains constant after the initial steps.The helper decoders needed to annotate weakly labeled replay images contribute to RECALL's storage requirements.
  • Ablation Study: Replay alone limits the 10-10-to-10-1 mIoU degradation to 8%, while combining replay and inpainting improves final results by up to 15%.Inpainting alone loses more than 20% across the same overlapped setups as incremental stages increase.
  • Ablation Study: mIoU is stable across new-to-replay sample ratios, with the maximum reached at rnew/rold = 1.This comparison evaluates distinct interleaving policies in the 15-1 disjoint setup.

8. Conclusions

The paper introduces RECALL for continual semantic segmentation, combining replay of old-class data with background inpainting. Experiments report improved accuracy and robustness across multiple incremental steps, while future work targets generative quality, weak supervision, and broader evaluation.

  • Conclusions: RECALL combines replay strategies with background inpainting to alleviate catastrophic forgetting and background shift in continual semantic segmentation.Replay data come from either a conditional GAN or web crawling.
  • Conclusions: The framework improves accuracy and robustness to multiple incremental steps compared with competing methods.The conclusion reports this as the outcome of the experimental analyses.
  • Future Work: Future work includes improving the generative model, refining weak supervision during web crawling, and evaluating additional datasets such as ADE20K.

Supplementary Material

The supplementary material motivates RECALL's design choices and adds experiments concerning backbone pre-training and class-space conversion.

  • Supplementary Material: The supplement examines how the pre-training dataset used to initialize the ResNet101 backbone affects continual semantic segmentation performance.
  • Supplementary Material: It also discusses the Class Mapping Module that converts between the GAN and target class spaces.

S1. Analyses on Pre-Training

MS COCO pre-training consistently improves incremental semantic segmentation compared with ImageNet pre-training, with gains reaching 15%. The authors nevertheless avoid it because it may already encode information about classes introduced later.

  • MS COCO pre-training always improves incremental semantic segmentation across the considered setups.The comparison covers all experimental incremental protocols.
  • Up to 15% improvements are observed with MS COCO rather than ImageNet pre-training.The reported gap may also reflect the encoder-freezing procedure used in the experiments.
  • MS COCO pre-training can provide spatial and semantic information about classes added during later incremental steps.This is one reason the authors avoid the strategy despite its stronger results.
  • ImageNet pre-training requires more training steps to adapt because it comes from a different task and domain.MS COCO is closer to the target semantic-segmentation task and domain.

S2. Class Mapping Module

The Class Mapping Module translates target segmentation classes into suitable GAN classes so replay samples can be generated across datasets. Replay and background inpainting preserve old-class recognition more effectively than fine-tuning or inpainting alone.

  • Class Mapping Module: The Class Mapping Module uses a classifier’s summed class probabilities to map each target class to the highest-scoring GAN class.The mapping is needed because the target and GAN datasets have different label spaces.
  • Class Mapping Module: Only one unexpected pairing occurs: the VOC person class maps to ImageNet cowboy hat because ImageNet lacks person.Generated images for this pairing remain reasonable but show lower variability than the original VOC data.
  • Quantitative and qualitative effects: Fine-tuning catastrophically forgets previous classes, while background inpainting alone only partly alleviates forgetting.Inpainting outputs remain noisy and can lose object parts while overestimating background.
  • Quantitative and qualitative effects: Replay with GAN or Web data improves old-class preservation and reduces false positives on newly introduced classes.The replay strategies are combined with background-content inpainting to reduce bias toward background.
  • Qualitative evaluation: Qualitative evaluations cover six disjoint protocols and show replay outputs approaching the joint-training reference except near object boundaries.The protocols are 19-1, 15-5, 15-1, 10-10, 10-5, and 10-1.
  • Incremental-step behavior: In the 15-1 setup, replay preserves cow and bus recognition when visually similar sheep and train classes are added.Both GAN and Web replay correct confusions that persist with fine-tuning or inpainting alone.

S5. Combining RECALL with Competitors

RECALL’s replay framework can be added to competing continual-learning approaches rather than used only as a standalone method. Naive replay already helps, while the complete method yields larger improvements.

  • Replay with GAN-generated or web-crawled data is presented as a general framework that can augment competing approaches.The authors describe these sources as previously unused in continual semantic segmentation.
  • Naive pseudo-labeling with the previous decoder already improves mIoU on VOC2012 disjoint 15-1.In the comparison, G denotes GAN replay and F denotes Flickr replay.
  • Combining replay with the complete RECALL method produces improvements ranging from 17% to 46%.The comparison is reported for multiple competing approaches and supports the modules’ general applicability.

S6. Preliminary Analyses on ADE20K

Preliminary ADE20K experiments test RECALL beyond object-level VOC classes, including many stuff categories. In the 100-10 setting, GAN replay substantially exceeds fine-tuning on mIoU.

  • Dataset and setting: ADE20K extends the evaluation to a large class set containing stuff categories absent from Pascal VOC2012.The authors use it to examine continual segmentation beyond object-level classes.
  • Dataset and setting: RECALL can retrieve semantically correlated categories when exact class correspondence is unavailable.The mapping is automatic, and Web replay allows direct control over search keywords.
  • Results: In ADE20K’s 100-10 setting, RECALL with GAN replay reaches 11.4% mIoU versus 0.8% for fine-tuning.The setting learns 100 classes initially and adds the remaining classes in batches of 10.
  • Supplementary analyses: The supplementary evaluation includes qualitative figures and tables for replay samples, incremental outputs, class mappings, and per-round accuracy measures.The listed materials cover Figures S1-S3 and Tables S1-S5.
Loading 2108.03673v2…