PaperlayerView exact revision on arXiv

Source-linked AI summary

RECALL: Replay-based Continual Learning in Semantic Segmentation

Andrea Maracani, Umberto Michieli, Marco Toldo, Pietro Zanuttigh

arXiv:2108.03673v2cs.CV

TL;DR

Continual semantic segmentation methods can struggle with forgetting across multiple incremental steps and with shifts in the background class. RECALL combines replay of old-class data with background self-inpainting, and reports stable mIoU across increasingly complex disjoint setups.

  • Problem

    Existing continual semantic segmentation approaches can partially alleviate forgetting but often fail across multiple incremental steps or when the background distribution shifts.

    Source: §1 · p. 1

  • Method

    RECALL re-creates old-class samples using conditional GAN generation or web retrieval, mixes them with current training data, and uses background self-inpainting to address background shift.

    Source: §1 · p. 1, §4 · p. 3, §8 · p. 8

  • Results

    61.9%, 60.6%, and 57.8% mIoU across disjoint 10-10, 10-5, and 10-1 setups; RECALL exceeds competitors by about 8% in 10-5 and 37% in 10-1.

    Source: §7 · p. 7

  • Takeaways & Limitations

    The authors conclude RECALL improves accuracy and robustness to multiple incremental steps compared with competitors.

    Source: §8 · p. 8

    The web-retrieval variant requires internet access, uses additional almost-unsupervised training data, and offers no control over the web source's weak labeling.

    Source: §5 · p. 5

Abstract

from arXiv · show

Deep networks allow to obtain outstanding results in semantic segmentation, however they need to be trained in a single shot with a large amount of data. Continual learning settings where new classes are learned in incremental steps and previous training data is no longer available are challenging due to the catastrophic forgetting phenomenon. Existing approaches typically fail when several incremental steps are performed or in presence of a distribution shift of the background class. We tackle these issues by recreating no longer available data for the old classes and outlining a content inpainting scheme on the background class. We propose two sources for replay data. The first resorts to a generative adversarial network to sample from the class space of past learning steps. The second relies on web-crawled data to retrieve images containing examples of old classes from online databases. In both scenarios no samples of past steps are stored, thus avoiding privacy concerns. Replay data are then blended with new samples during the incremental steps. Our approach, RECALL, outperforms state-of-the-art methods.

1. Introduction · 2. Related Works

RECALL tackles catastrophic forgetting in class-incremental semantic segmentation by replaying recreated examples of old classes and using background inpainting to mitigate background shift. It draws on both a pretrained GAN and web-crawled images, while related work situates it among continual-learning replay methods and webly supervised learning.

1. Introduction

  • Incremental segmentation suffers severe catastrophic forgetting; distillation-based approaches can fail across multiple steps or when background statistics shift as classes change.Background shift occurs when the background incorporates old or future classes across learning steps.Source: §1 · p. 1
  • Replay uses either a pretrained GAN conditioned on old classes or web-crawled images queried by class name, then assigns segmentation pseudo-labels with a minimally stored side module.The GAN was trained on a different dataset, ImageNet, so a Class Mapping Module translates between label spaces; both sources provide weakly labeled data.Source: §1 · pp. 1–2
  • RECALL replaces teacher-to-student distillation with replay of recreated old-class representations mixed with new training data, and reassigns background regions using previous-model predictions.The paper presents replay as the first such approach for continual semantic segmentation and introduces self-inpainting to address background shift.Source: §1 · p. 2
  • The paper claims RECALL is the first continual-segmentation replay method and first to use webly supervised replay, and reports state-of-the-art results across scenarios, especially multiple incremental steps.It also presents background inpainting as a strategy for generating pseudo-labels and overcoming background shift.Source: §1 · p. 2

2. Related Works

  • Continual-learning methods include dynamic architectures, regularization, rehearsal, and generative replay; RECALL uses replay without storing past samples or training a GAN on target data.Unlike cited image-classification approaches, it does not select real exemplars as anchors or train or fine-tune the GAN on the current distribution [40] [44].Source: §2 · p. 2
  • Prior class-incremental segmentation work explores freezing and distillation [30], class-importance weighting [23], background shift [5], spatial relationships [11], and latent-space regularization [31].These works address learning new categories from new data; earlier segmentation networks commonly used auto-encoder structures beginning with FCN [28].Source: §2 · p. 2
  • Webly supervised learning uses weakly labeled web images or videos in segmentation, but the paper identifies its use as a continual-learning replay strategy as unexplored.Prior research had focused on querying, filtering, and exploiting web data, including assigning pseudo-labels.Source: §2 · p. 2

3. Problem Formulation · 4. General Architecture

RECALL addresses class-incremental semantic segmentation, where only data for newly added classes is available at each step. It combines replay of past classes with background self-inpainting to mitigate forgetting and background shift.

3. Problem Formulation

  • Each pixel is assigned a semantic class, but at step k only data for new classes Ck is available; RECALL keeps encoder E0 fixed and updates the decoder.The model’s output space expands as new, disjoint class sets are added.Source: §3 · pp. 2–3
  • RECALL addresses both disjoint and overlapped setups: disjoint steps use separate training partitions, while overlapped steps may reuse images containing labeled pixels from new classes.In both setups, only the current step’s classes are annotated; other pixels are assigned to background.Source: §3 · p. 3
  • The background class undergoes semantic shift at each step because pixels from changing sets of classes are assigned to background.This shift occurs in both disjoint and overlapped setups.Source: §3 · p. 3

4. General Architecture

  • At each incremental step, background regions in current labels are replaced with the previous model’s predictions, while step 0 is left unchanged because no prior model is available.This self-inpainting transfers past knowledge to background regions and also provides a regularizing effect similar to knowledge distillation, though implemented differently.Source: §4 · pp. 3–4, Figure 3 · p. 4
  • RECALL retrieves past-class images with a conditional GAN or web crawler, labels them with a Label Evaluation Block, and combines replay data with current data for incremental training.The Source Block supplies images without pixel-level labels; the Label Evaluation Block provides the missing annotations.Source: §4 · p. 3, Figure 2 · p. 4
  • Replay examples are generated and labeled for past class sets, then added to current-step data; replay-augmented training mitigates bias toward new classes and helps prevent forgetting.At step k, past incremental class sets are sampled uniformly for replay.Source: §4 · p. 4
  • Training interleaves rnew current-partition samples with rold replay samples per batch, and their ratio controls the balance between learning new classes and recalling past knowledge.The authors use this policy rather than random sampling to ensure a proper stream of information.Source: §4 · p. 5

5. Replay Strategies

The section describes two replay sources: a conditional GAN that regenerates examples of previously seen classes and web-crawled images, with replay images passed to the Label Evaluation Block for annotation.

Replay by GAN

  • The GAN replay strategy uses a conditional GAN pretrained on ImageNet to regenerate unavailable examples, mapping target classes to statistically similar GAN classes.The GAN is pretrained on a generic large-scale visual dataset; its classes and domains need not exactly match the segmentation dataset.Source: §5 · p. 5
  • A pretrained classifier maps each target class by aggregating softmax scores over its training subset, selecting the top-scoring GAN class and retaining mappings across steps.Source: §5 · p. 5
  • At each incremental step, mapped classes from past iterations condition GAN generation, and the Label Evaluation Block assigns labels to the resulting replay images.The resulting self-annotated data supports supervised training on past classes alongside the current step.Source: §5 · p. 5

Replay by Web Crawler

  • The web crawler retrieves images whose tags and descriptions both contain a past class name, then sends them to the Label Evaluation Block for annotation.Flickr was used in the evaluation, though other online databases or search engines can be used.Source: §5 · p. 5
  • Unlike GAN replay, online retrieval adds no learnable modules or need for a larger diverse dataset, but requires internet access and offers little control over weak labels.It also uses additional training data, albeit almost unsupervised.Source: §5 · p. 5

6. Implementation Details

RECALL uses an ImageNet-pretrained DeepLab-V2/ResNet-101 setup while remaining architecture-independent, with different training schedules for the main and helper decoders. Its replay implementation uses BigGAN-deep samples mapped to segmentation classes and combines them with new samples at a 1:1 ratio.

6. Implementation Details

  • RECALL uses DeepLab-V2 with ResNet-101 [18] but is architecture-independent; the ImageNet-pretrained encoder is trained at step 0, while later steps train the main and helper decoders.Helper decoders annotate replay samples.Source: §6 · pp. 5–6
  • SGD uses momentum and a 5 × 10−4 to 5 × 10−6 polynomial schedule; training lasts |Ck|×1000 steps for disjoint and |Ck| × 1500 for overlapped setups [30].Helper decoders use a separate 2 × 10−4 to 2 × 10−6 polynomial schedule for |Ck| × 1000 steps.Source: §6 · p. 6
  • At each incremental step, RECALL generates 500 replay samples per old class from ImageNet-pretrained BigGAN-deep [4], maps classes with an ImageNet-pretrained EfficientNet-B2 classifier [42], and uses a 1:1 old-to-new sample ratio.Source: §6 · p. 6
  • Inputs are randomly scaled and mirrored, then padded and cropped to 321 × 321 px; the TensorFlow [3] framework runs on one NVIDIA RTX 2070 Super, with the longest training run taking about 5 hours.Code and replay data are available at the cited repository.Source: §6 · p. 6

7. Experimental Results

On Pascal VOC 2012, RECALL is evaluated across incremental class-addition scenarios and compared with fine-tuning, joint training, replay, and continual-segmentation methods. The results show its replay approaches limit forgetting, especially in more challenging incremental settings, while ablations examine memory, module contributions, and replay balance.

7. Experimental Results

  • In 19-1, RECALL achieves higher overall mIoU than competitors; in 15-5 and 15-1, replay approaches restore past-class knowledge and approach joint-training mIoU.The FT-to-joint-training mIoU gap rises from about 41% in 19-1 to more than 70% in 15-1; S&R trails RECALL by more than 20% mIoU in the last-5-class scenarios.Source: §7 · p. 6
  • In disjoint 10-5 and 10-1, RECALL outperforms competitors by about 8% and 37% mIoU, respectively, while remaining comparable in 10-10.Across disjoint 10-10, 10-5, and 10-1, RECALL maintains 61.9%, 60.6%, and 57.8% mIoU; FT falls from 33.0% to 23.7% to 4.3%, and SDR from 62.9% to 52.0% to 20.8%.Source: Table 1 · p. 7, §7 · p. 7
  • Qualitative examples show RECALL preserving old-class objects and reducing novel-class bias, with web replay more accurate than GAN replay in one horse example.The examples include preserved bus and horse features and reduced replacement of old classes by newly added classes.Source: §7 · p. 7, Figure 5 · p. 8

7.1. Ablation Study

  • The web-based solution uses memory close to standard no-sample-storage approaches, while GAN storage is comparable to storing images initially but remains constant as image storage grows.The figure compares stored-data memory across incremental steps and learned-class counts.Source: §7.1 · pp. 7–8, Figure 6 · p. 8
  • Inpainting alone preserves knowledge but weakens over multiple steps; replay alone limits the 10-10-to-10-1 overlapped mIoU drop to 8%, and combining both boosts results by up to 15%.On that same overlapped transition, inpainting alone sees a drop of more than 20%; joint replay and inpainting improve final results in all setups.Source: §7.1 · p. 8, Table 2 · p. 8
  • mIoU is relatively stable across new-to-replay sample proportions and reaches its maximum when rnew/rold = 1.The interleaving-policy analysis is for the 15-1 disjoint setup.Source: §7.1 · p. 8, Figure 7 · p. 8

8. Conclusions

RECALL combines replay of former-stage data with background inpainting to address catastrophic forgetting and background shift in continual semantic segmentation. Experiments show improved accuracy and robustness to multiple incremental steps compared with competitors.

8. Conclusions

  • RECALL uses replay via a conditional GAN or web-crawled images, alongside background inpainting, to address catastrophic forgetting and background shift.The two replay schemes retrieve data related to former training stages without storing past samples.Source: §8 · p. 8
  • Experiments show RECALL improves accuracy and robustness to multiple incremental steps compared with competitors.Source: §8 · p. 8
  • Future work will improve the generative model, refine control of weak supervision during web crawling, and evaluate RECALL on datasets such as ADE20K.Source: §8 · p. 8

RECALL: Replay-based Continual Learning · Supplementary Material

The supplementary material motivates RECALL’s design choices and reports additional experiments, covering backbone pre-training, class mapping, benchmark analyses, and combinations with competing methods.

Supplementary Material

S1. Analyses on Pre-Training

MS COCO pre-training improves incremental semantic segmentation, but RECALL uses ImageNet pre-training in the main paper to avoid possible overlap with classes learned incrementally. The supplementary analysis attributes the gains to task-relevant initialization and semantic transfer, with the gap varying by setup.

Analyses on Pre-Training

  • The main paper conservatively uses ImageNet pre-training because MS COCO pre-training could include information about classes learned in incremental steps, following prior works [30] [31] [5].ImageNet is a common classification pre-training source, while MS COCO is a related segmentation dataset [27].Source: S1. Analyses on Pre-Training · p. 11
  • MS COCO pre-training improves incremental semantic segmentation in every considered setup, with improvements of up to 15%.The comparison is with ImageNet pre-training.Source: S1. Analyses on Pre-Training · p. 11
  • MS COCO may help through better initialization for the same task and prior spatial and semantic learning of classes introduced incrementally.The latter possibility is why the authors avoid this strategy despite its better results.Source: S1. Analyses on Pre-Training · p. 11
  • Encoder freezing can reduce catastrophic forgetting but prevents new-class samples from updating feature extraction, restricting their updates to the decoder.The authors suggest this may contribute to the large pre-training gap.Source: S1. Analyses on Pre-Training · p. 11
  • The mIoU gap between MS COCO and ImageNet pre-training is larger when the initial step contains fewer classes, such as 10.Source: S1. Analyses on Pre-Training · p. 11

S2. Class Mapping Module · S3. Per-Class Quantitative Results

The Class Mapping Module uses classifier-based matching to connect segmentation classes with GAN-training classes, while per-class results show that replay and background inpainting support knowledge preservation and improve performance on newly added classes.

S2. Class Mapping Module

  • The module maps each segmentation class to a GAN-training class by summing a pretrained classifier’s probability vectors over that class’s training samples and selecting the argmax.Correct cross-domain label pairings matter because incorrect matches may harm final-model accuracy.Source: S2. Class Mapping Module · p. 11
  • All mappings appeared reasonable except VOC person→ImageNet cowboy hat; because ImageNet lacks person, this proxy was considered plausible across domains and non-nested label sets.The cowboy hat was considered a reasonable match because it is shown on top of people’s heads.Source: S2. Class Mapping Module · p. 11
  • Generated images for person mapped to cowboy hat look reasonable, though their variability is lower than that of original VOC person-class data.The passage points to Figure S1 for visual examples.Source: S2. Class Mapping Module · pp. 11–12

S3. Per-Class Quantitative Results

  • Fine-tuning catastrophically forgets previous classes, while GAN or Web replay with background inpainting preserves old knowledge and improves new-class mIoU by reducing false positives.Background inpainting alone largely alleviates forgetting, with an effect similar to knowledge distillation [5].Source: S3. Per-Class Quantitative Results · p. 12
  • Fine-tuning achieves high pixel accuracy but low IoU on the added class, indicating overestimation; RECALL instead maintains steady mIoU across incremental steps.Fine-tuning also has very low mIoU across all learned classes and previously seen categories, whereas high pixel accuracy reflects frequent new-class predictions despite false positives.Source: S3. Per-Class Quantitative Results · p. 12

S4. Additional Qualitative Results · S5. Combining RECALL with Competitors

The supplementary results show that replay preserves old-class recognition in qualitative examples, while combining RECALL with competing approaches improves performance. They also illustrate generated images and segmentation outputs across incremental setups and steps.

S4. Additional Qualitative Results

  • Figure S2 shows segmentation outputs for all five training protocols across six disjoint setups, with examples from each setup.The protocols are fine-tuning, background inpainting, RECALL with GAN or Web, and joint training; the setups are 19-1, 15-5, 15-1, 10-10, 10-5, and 10-1.Source: S4. Additional Qualitative Results · p. 12
  • Generated GAN images are usually similar to Pascal VOC images, including person examples without a direct class mapping, and are shown alongside Pascal VOC and Flickr samples.Figure S1 includes two randomly sampled images per class from Pascal VOC2012 or Flickr for comparison.Source: S4. Additional Qualitative Results · p. 12
  • In qualitative disjoint-scenario examples, GAN and Web replay reduce forgetting and yield segmentation close to joint training, while inpainting alone remains noisy and can lose object parts.Replay also avoids confusing old classes with similar new ones, unlike fine-tuning and inpainting, which mistake cow for sheep and bus for train in examples.Source: S4. Additional Qualitative Results · p. 12
  • Across the 15-1 incremental steps, GAN and Web replay preserve cow and bus recognition when sheep and train are introduced, correcting confusions seen with fine-tuning or inpainting.Figure S3 shows output maps after each step for two image samples.Source: S4. Additional Qualitative Results · p. 12

S5. Combining RECALL with Competitors

  • The section presents RECALL as a framework for retrieving and using unlabeled replay data that can be applied on top of competing continual-segmentation approaches.The authors state that, to their knowledge, no prior continual semantic-segmentation work uses GAN-generated or web-crawled data.Source: S5. Combining RECALL with Competitors · p. 12
  • Combining RECALL with competing approaches yields improvements ranging from 17% to 46%, demonstrating the modules’ effectiveness and general applicability.The section presents the framework as a way to retrieve and employ unlabeled replay data; Table S2 compares replay with naïve pseudo-labeling and the complete RECALL approach on disjoint 15-1.Source: S5. Combining RECALL with Competitors · pp. 12–13, Table S2 · p. 13

S6. Preliminary Analyses on ADE20K

Preliminary ADE20K experiments suggest RECALL can mitigate forgetting in a setting with many classes, including stuff categories beyond Pascal VOC2012. Its replay strategy can retrieve classes through semantically related categories even when the generative model is not conditioned on them.

Preliminary results

  • ADE20K poses a challenge beyond Pascal VOC2012 because it has a vast class set that includes stuff categories as well as object-level classes.The authors use this difference to motivate preliminary experiments in the 100-10 incremental setting.Source: S6. Preliminary Analyses on ADE20K · p. 13

Replay for ADE20K

  • RECALL automatically maps semantically correlated generated categories to segmentation classes absent from the GAN’s conditioning space; web replay also allows direct control over search keywords.The paper gives retrieving people from generated images of related categories such as hats as an example.Source: S6. Preliminary Analyses on ADE20K · p. 13

Preliminary results

  • In the 100-10 ADE20K setting, RECALL with GAN-based replay reached 11.4% mIoU versus 0.8% for FT.The experiments used the same preprocessing, learning parameters, batch size, and image size as VOC2012, without parameter tuning; further experiments and tuning are needed for a proper benchmark comparison.Source: S6. Preliminary Analyses on ADE20K · p. 13

Related papers

Loading 2108.03673v2…