PaperlayerView exact revision on arXiv

Source-linked AI summary

SELECT: SELEctive Context Transfer for Class-Incremental Semantic Segmentation

Avi Gupta, Saurabh Yadav, Koteswar Rao Jerripothula, Tammam Tillo

arXiv:2608.30281v1cs.CV

TL;DR

Class-Incremental Semantic Segmentation must learn new classes while retaining previously learned ones, but background initialization and global transfer can provide ambiguous or diluted knowledge. SELECT selectively anchors each new class to semantically similar past classes, using Context Transfer Attention plus perturbation and margin-based separation. The method reports state-of-the-art performance across most Pascal VOC and ADE20K settings and improves stability-plasticity balance through selective knowledge transfer.

  • Problem

    CISS must learn new classes while retaining past knowledge, but background initialization and global distillation can provide noisy or diluted semantic information.

  • Method

    SELECT identifies semantically similar past classes, aggregates their tokens into a new-class initialization with Context Transfer Attention, and uses perturbation plus a margin loss to separate source and new representations.

  • Results

    SELECT achieves state-of-the-art performance across most Pascal VOC and ADE20K settings.

  • Takeaways & Limitations

    Selective transfer provides a more principled way to use prior knowledge while preserving distinctiveness between new and influential old-class representations.

Abstract

from arXiv · show

Class-Incremental Semantic Segmentation (CISS) is fundamentally challenged by catastrophic forgetting and background shift, where learning new concepts degrades performance on previously seen classes. While existing methods attempt to balance stability (retaining old knowledge) and plasticity (learning new knowledge), they often fail to leverage prior knowledge effectively. These approaches typically rely on indiscriminate knowledge transfer or ambiguous initializations, which can dilute crucial semantic information. To overcome this limitation, we propose SELECT, a novel approach for Selective Context Transfer, which instead grounds each new class in a small set of semantically similar past classes. Its core is a Context Transfer Attention mechanism that aggregates the learned tokens from similar classes into a structured initialization for the new class. To ensure this transfer does not corrupt the borrowed representations, we add a controlled noise perturbation and a margin-based context-transfer loss that enforces separation between the new class token and its source tokens. Extensive experiments on Pascal VOC and ADE20K show that SELECT consistently outperforms prior work, achieving mIoU of 2.2% on VOC and 2.8% on ADE, providing an effective handle on the stability-plasticity dilemma. Code is available at https://github.com/avigupta2798/SELECT.

2 Microsoft, India

The supplied passage identifies an affiliation in Kanpur, India.

  • The affiliation is listed as IIT Kanpur, Kanpur, India.

1 Introduction

Class-Incremental Semantic Segmentation requires learning new classes while retaining prior knowledge, but catastrophic forgetting and ambiguous background-based initialization hinder this balance. SELECT addresses these issues by selectively transferring context from semantically similar past classes through structured token initialization and separation-aware training.

  • Class-Incremental Semantic Segmentation requires models to learn new classes while retaining knowledge of previously seen classes.
  • Catastrophic forgetting forces CISS models to balance stability, plasticity, and intelligent knowledge transfer.
  • Background-based initialization provides a noisy, semantically ambiguous starting point that can impair new-class learning.
  • SELECT retrieves a small subset of semantically similar past classes and uses their tokens to initialize each incoming class.
  • Context Transfer Attention aggregates relevant source tokens into a structured initialization that supports knowledge transfer while promoting plasticity.
  • Controlled noise and a margin-based transfer loss help separate the new token from source tokens and protect borrowed representations.

2 Related Works

Related work develops methods for retaining knowledge during incremental learning, including regularization, architectural changes, distillation, and prior reuse. In semantic segmentation, these approaches address catastrophic forgetting and background shift but remain part of an evolving research area.

  • Class-incremental learning methods use regularization to limit parameter changes and preserve previous-task output distributions.
  • Architectural approaches represent another strategy for reducing catastrophic forgetting during incremental learning.
  • Class-incremental semantic segmentation extends incremental learning to dense prediction while retaining previously learned classes.
  • MiB addresses background shift through knowledge distillation, while earlier work also explored knowledge reuse in related settings.
  • Deep neural network approaches have faced representation interference when handling sequences of similar and dissimilar tasks.

3 Methodology

SELECT formulates CISS as sequential learning over disjoint class sets and replaces ambiguous initialization with selective transfer from semantically relevant past classes. It identifies related classes, constructs an informed new-class representation, and uses perturbation to preserve separation from source representations.

  • Problem Formulation: CISS trains an encoder-decoder sequentially across disjoint task-specific class sets while retaining performance on the cumulative label space.
  • Analysis on Efficient Knowledge Transfer: Random, background-based, and flawed-distillation initializations provide vague starting directions that can hinder convergence.Background representations are described as a catch-all for undefined objects, while prior-class reuse can select features collapsed into background noise.
  • Context-driven Selective Knowledge Transfer: Context Transfer Attention aggregates tokens from selected source classes into an adaptive token for the incoming class.The method focuses on semantically relevant classes rather than transferring initial knowledge entirely or mostly from the background.
  • Identifying Similar Classes: SELECT identifies a subset of semantically relevant past classes and transfers their learned representations to initialize each new class.The previous model probes masked new-class images, measures representation distances, and retains classes selected often enough under threshold ε.
  • Enforcing Representational Separation: If the transferred token is too close to source tokens, representations can merge and damage old-class decision boundaries.The paper identifies this overlap as a risk for catastrophic forgetting or confusing old classes with new ones.
  • Enforcing Representational Separation: Controlled Gaussian perturbation balances preserving transferred knowledge with introducing diversity around the adaptive token.The mixing coefficient α controls the trade-off between alignment and separation.

4 Experiments

Experiments evaluate SELECT across incremental settings, datasets, similarity choices, initialization strategies, and loss components. SELECT generally outperforms prior methods while balancing old-class stability with new-class plasticity.

  • State-of-the-art comparison: SELECT outperforms existing baselines across Pascal VOC and ADE20K training setups, including comparisons with a prior knowledge-transfer method.The authors also provide backbone-matched comparisons for fair evaluation.
  • Long task sequences: SELECT performs better than MBS across longer task sequences and shows a lower performance-decline rate in ADE20K 100-5.The evaluated sequences include ADE20K 100-5 and 100-10 and Pascal VOC 5-3.
  • Class similarity: Using similar rather than dissimilar past classes provides a more effective foundation for adapting to new classes while preserving prior knowledge.Transferring from dissimilar classes substantially harms both fundamental- and incremental-class performance.
  • Similarity metrics: Euclidean distance consistently outperforms alternative similarity metrics, particularly for incremental tasks.It measures both magnitude and orientation when comparing learned class tokens with image-specific class representations.
  • Initialization and loss: Replacing background initialization with CTA improves performance by approximately 9% on 15-1 and 1% on 15-5, while adding Lct contributes approximately 4% and 2%.CTA provides a structured starting point, and Lct enforces a minimum margin between the new token and similar source tokens.
  • Loss ablations: Removing Lfd reduces 15-1 new-class mIoU from 72.0% to 14.2%, while removing Lkd reduces it to 32.6%.Removing Lct lowers overall mIoU from 80.5% to 76.1% and incremental-class mIoU from 72.0% to 60.7%.
  • Knowledge transfer: CTA improves knowledge transfer over selecting one class or averaging all similar classes, achieving at least 11% and 41% improvements on the 1-15 and 16-20 tasks.The ablation also indicates that attention and controlled noise are complementary: either component alone can cause class-boundary problems.
  • Stability-plasticity: At the final 15-1 step, SELECT reaches 83.3% old-class mIoU and 72.0% new-class mIoU, exceeding MBS, NeST, and BARM on both dimensions.MBS reaches 82.3% and 69.0%, NeST 76.8% and 54.4%, and BARM 68.3% and 27.2% for old- and new-class mIoU, respectively.

Conclusion

SELECT addresses adaptive knowledge transfer by identifying semantic similarities between old and new classes and selectively transferring knowledge. Its loss function encourages distinctiveness between transferred knowledge and influential old-class representations.

  • SELECT identifies semantic similarities between old and new classes to transfer knowledge selectively.
  • Its loss function encourages distinctiveness between transferred knowledge and the original representations of influential old classes.
  • Empirical results demonstrate the effectiveness of the proposed approach.

A Complete Training Pipeline

SELECT’s training pipeline identifies similar previously learned classes and selectively transfers their knowledge to initialize the new class. Context Transfer Attention aggregates corresponding source tokens before incremental training.

  • The pipeline uses current-task data and the previous task’s model to identify similar classes.
  • Corresponding tokens from similar previous classes are aggregated with Context Transfer Attention.
  • The resulting initial class representation initializes the current task’s model before incremental training.
  • The current-task model is trained until convergence in the incremental setting.

B Additional Implementation Details

Experiments were conducted in a unified PyTorch environment with reimplemented baselines for fair comparison. Additional analysis visualizes representational perturbations and reports within-class similarity patterns.

  • Experiments used PyTorch 1.10.1 on a single workstation with an NVIDIA A100 GPU.
  • SELECT and MBS were trained and evaluated in the same unified environment for fair comparison.
  • Results for several comparison methods were reimplemented using their official publicly available source codes without modifying core architectures or losses.
  • Visualized perturbations show deviation within classes, with some classes exhibiting minimal deviation and strong characteristic similarity.

C.2 Further Analysis on Semantic Similarity

Additional analyses show that semantically similar initialization substantially improves incremental-class performance and supports retention of base-class representations. Processing comparisons also report higher speed for SELECT than NeST across ADE20K settings.

  • Further Analysis on Semantic Similarity: 84.22% mIoU versus 42.28% shows that similar initialization improves Sheep performance across eight configurations.
  • Further Analysis on Semantic Similarity: Semantically similar initialization lets the decoder adapt with fewer gradient steps.
  • Further Analysis on Semantic Similarity: Initialization-source choice also affects how well base classes are retained.
  • Further Analysis on Semantic Similarity: SELECT leverages a representational space in which features are semantically clustered, supporting selective class choices.
  • Additional Implementation Details: 70.0/10.2 imgs/sec on 100-50, 69.4/10 on 50-50, 71.6/11.1 on 100-10, and 74.0/13 on 100-5 compare SELECT with NeST.

D.1 Diverse Challenging Scenarios

Additional experiments show SELECT remains competitive across overlapped and disjoint class-incremental settings on Pascal VOC and ADE20K, including more challenging scenarios and incremental-class analyses.

  • Quantitative comparisons: 78.6% All setting mIoU in the 5-3 overlapped Pascal VOC setting exceeds MBS’s 77.0%.The 5-3 setting challenges SELECT with only five base classes and fewer potential similar classes.
  • Quantitative comparisons: SELECT provides a strong competitive advantage over previous approaches across disjoint Pascal VOC settings of 19-1, 15-5, and 15-1.These settings cover two-task and six-task incremental scenarios.
  • Quantitative comparisons: SELECT surpasses prior work across almost all evaluated ADE20K overlapped settings.ADE20K is presented as a more diverse and robust-scene benchmark.
  • Class-wise analysis: SELECT and MBS perform similarly on fundamental classes, but MBS shows a performance drop across incremental classes in all three Pascal VOC settings.The comparison covers the 15-1, 15-5, and 19-1 settings.
  • Experimental scope: Tables 8–10 report comparisons across Pascal VOC overlapped and disjoint scenarios and ADE20K overlapped scenarios.The tables include reproduced or excerpted results where indicated by their captions.

D.3 Additional Qualitative Results

Qualitative comparisons and supplementary analyses show SELECT retaining prior-class knowledge while adapting to new classes, with performance supported across backbones and hyperparameter choices.

  • Qualitative comparisons: SELECT produces fewer accumulating false positives than MBS while preserving previously learned objects as incremental tasks progress.In the example, the sheep learned in the third task remains represented thereafter.
  • Qualitative comparisons: SELECT achieves closer pixel-wise similarity to ground truth than REMINDER and NeST in Pascal VOC’s 15-5 final segmentation results.The comparison is presented as evidence of robustness.
  • Backbone robustness: On the 15-1 setting, SELECT improves over standard MiB by approximately 20% and over PLOP by approximately 15% across all classes with CNN backbones.The method was integrated into MiB and PLOP using Res101 backbones.
  • Backbone robustness: SELECT significantly outperforms NeST across the 15-1, 15-5, and 10-1 tasks under the ViT-B backbone evaluation.The reported results are averaged across all classes.
  • Hyperparameter analysis: Euclidean similarity performs best among the evaluated metrics in both Pascal VOC 15-1 and 15-5 settings.The analysis states that Euclidean distance considers both magnitude and orientation.
  • Hyperparameter analysis: σ=0.05 provides the best performance across both noise-analysis tasks, while smaller and larger values under-diversify or over-perturb initialization.The comparison evaluates σ=0.01 and σ=0.10 as examples of the two failure modes.
  • Hyperparameter analysis: A context-transfer margin of M=1.0 performs best, whereas M=0.0 fails to enforce a distinct decision boundary.The margin controls separation between the new class and similar old classes.
  • Hyperparameter analysis: A frequency threshold of 0.15 performs best, while larger thresholds introduce semantic dilution by including more similar classes.The comparison reports declines at thresholds 0.30 and 0.45.
Loading 2608.30281v1…