PaperlayerView exact revision on arXiv

Source-linked AI summary

Selective Backpropagation for Efficient Few-Shot Class-Incremental Learning

Eeham Khan, Abdulmoumen Al-Atrash, Ali Ayub

arXiv:2610.04003v1cs.LGcs.AIcs.CV

TL;DR

FSCIL must balance retaining prior classes with learning new ones from scarce labels, yet short, distribution-consistent benchmarks may not reveal behavior under distribution shift or long horizons. The paper proposes Selective Backpropagation, which budgets trainable parameters across sessions; it reports strong benchmark and extended-setting performance with substantially reduced training cost.

  • Problem

    FSCIL requires learning novel classes from scarce labels while retaining prior knowledge without replay, and short, distribution-consistent benchmarks may not capture behavior under shift or long horizons.

    Source: §1 · pp. 1–2

  • Method

    Selective Backpropagation assigns parameters to current learning, freezes parameters associated with past tasks, and reserves and reinitializes free parameters for future tasks.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    SBP achieves state-of-the-art performance across multiple FSCIL benchmarks and remains strong across cross-domain shifts and an 80-session ImageNet-1K stream while substantially reducing training cost.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The reported results indicate that standard short benchmarks alone are insufficient to characterize FSCIL behavior across distribution shifts and longer learning horizons.

    Source: §3.3 · p. 8, §5 · p. 10

    Evaluation on short, distribution-consistent benchmarks alone is insufficient to characterize behavior under distribution shift or over longer learning horizons.

    Source: §3.3 · p. 8, §5 · p. 10

Abstract

from arXiv · show

Few-Shot Class-Incremental Learning (FSCIL) requires models to continuously learn new classes from limited samples while retaining prior knowledge, under strict constraints on compute and memory. Existing approaches lie along a difficult trade-off: simple fine-tuning is computationally efficient but suffers from catastrophic forgetting, replay-based methods mitigate forgetting at the cost of substantial compute and memory, and exemplar-free methods often reduce forgetting by freezing most of the backbone, improving efficiency at the expense of adaptability. We propose Selective Backpropagation (SBP), a deterministic parameter budgeting framework that bridges this gap. SBP restricts gradient updates to a pre-allocated subset of network parameters, freezing past knowledge and preserving unbiased capacity for future learning, enabling rapid adaptation without costly mask optimization. We show that SBP achieves strong performance across standard FSCIL benchmarks while requiring training time close to that of naive fine-tuning and substantially lower training time than prior SOTA methods. Crucially, our experiments expose a limitation of standard FSCIL evaluation: performance on short, distribution-consistent benchmarks does not necessarily predict behavior under distribution shift or over substantially longer learning horizons. We therefore evaluate FSCIL methods in cross-domain settings and over an 80-session ImageNet-1K stream. SBP remains strong across these regimes while maintaining low training cost, providing a favorable stability-plasticity-efficiency trade-off. Our code is available at https://github.com/PaInt-Lab/sbp-main-public.

1 INTRODUCTION

FSCIL asks models to learn new classes from scarce labels while retaining prior knowledge, without replay and under compute constraints. The paper proposes SBP as a parameter-budgeting approach and argues that evaluation must extend beyond short, distribution-consistent benchmarks.

1 INTRODUCTION

  • FSCIL requires retaining prior classes without replay while learning novel classes from extremely limited supervision, creating a stability–plasticity tension.This challenge is compounded by an accuracy–efficiency trade-off.Source: §1 · p. 1
  • Short, distribution-consistent benchmarks may not predict adaptation under distribution shift or interference over longer learning horizons.The paper argues evaluation should jointly test retention, adaptation, longer horizons, and computational cost.Source: §1 · pp. 1–2
  • Fine-tuning is fast and adaptive but forgets prior classes, while replay or regularization adds overhead and backbone freezing can sacrifice plasticity.Parameter-isolation methods offer a potential compromise but often require costly parameter search or learned masks, and many rely on task identity at test time.Source: §1 · p. 2
  • SBP preallocates network capacity to balance stability and efficiency under exemplar-free FSCIL constraints.It freezes parameters for past knowledge, assigns parameters to current learning, and reserves free parameters for future tasks.Source: §1 · p. 2
  • SBP reports state-of-the-art performance on CIFAR-100 and CUB-200, competitiveness on MiniImageNet, and strong cross-domain and long-horizon results at lower training cost.The introduction presents these results as a stability–plasticity–efficiency balance for realistic FSCIL scenarios.Source: §1 · p. 2

2 METHODOLOGY

The methodology defines FSCIL as session-based learning with few labeled examples for each novel class and limited access to data. SBP allocates trainable capacity across sessions, reinitializes unused capacity, and pairs the resulting features with prototype-based classification.

2 METHODOLOGY

  • Each incremental session introduces novel classes with only a few labeled examples per class, and only that session’s data is accessible during adaptation.The base session uses abundant labels; deployment also imposes limited training time and restricted hardware resources.Source: §2 · p. 2

2.1 SELECTIVE BACKPROPAGATION

  • SBP partitions parameters into current-task assigned, past-task frozen, and future-task free subsets, with gradients restricted to assigned parameters.Assignment is irreversible: capacity moves from free to assigned to previous, and past parameters are not reassigned or further optimized.Source: §2.1 · p. 3
  • A larger allocation ratio is used for the data-rich base session, while later sessions use a constant allocation ratio that can be small.The passage gives approximately 1% of the network per incremental session as an example; freezing past parameters preserves learned representations and supports transfer.Source: §2.1 · pp. 3–4

2.2 FREE PARAMETER REINITIALIZATION

  • Free parameters are reinitialized each epoch and receive no gradients, preventing current tasks from depending on capacity reserved for later sessions.Their changing values make the free subspace non-stationary during optimization, so the learned representation does not rely on it in practice.Source: §2.1 · p. 3, §2.2 · p. 4

2.3 PROTOTYPE-BASED CLASSIFICATION

  • New-class weights are initialized from class-mean feature prototypes, and classification compares prototypes only over dimensions allocated through the current session.Stored prototypes are frozen after each increment; restricting distance computation to the active subspace addresses possible mismatch with changing feature representations.Source: §2.3 · p. 4

3 EXPERIMENTS

Experiments compare FSCIL methods on standard benchmarks, under cross-domain shift, and across an 80-session ImageNet-1K stream, measuring accuracy and training time.

3 EXPERIMENTS

  • The standard CIFAR-100 and MiniImageNet protocol uses 60 fully supervised base classes followed by eight 5-way 5-shot incremental sessions, with methods compared using ResNet-18 and five-run means.The evaluation also includes a MiniImageNet-to-CUB-200 cross-domain setting and an ImageNet-1K stream with 600 base classes and 400 novel classes across 80 sessions.Source: §3 · pp. 4–5, §3.2 · p. 7, §3.3 · p. 7

3.1 EXPERIMENTS ON CIFAR-100 AND MINIIMAGENET

  • On CIFAR-100, SBP achieves 57.90% final-session accuracy and 68.73% average incremental accuracy, each above the second best; on MiniImageNet, it reaches 54.10% and 65.26%, remaining highly competitive.On MiniImageNet, WaRP trains in 27 minutes versus SBP’s 30, but achieves approximately 10% lower final and average incremental accuracy.Source: §3.1 · pp. 5–6

3.2 CROSS-DOMAIN FSCIL

  • In cross-domain FSCIL, SBP leads every incremental session, reaching 38.57% final accuracy and 53.34% average accuracy, 2.45% and 3.21% above WaRP.Methods that freeze the backbone after the base session degrade substantially because their fixed representations cannot adapt to shifted incremental classes.Source: §3.2 · p. 7, Table 3 · p. 7

3.3 LONG-HORIZON FSCIL

  • Across 80 ImageNet-1K incremental sessions, SBP retains 39.08% accuracy, while WaRP falls from 51.40% at Session 30 to 12.90% at Session 80.ADBS and OrCo reach 14.50% and 18.40% at the final session; short FSCIL sequences do not necessarily predict long-horizon behavior.Source: §3.3 · pp. 7–8, Table 4 · p. 8

3.4 ABLATION STUDY

  • Removing SBP drops average incremental accuracy from 68.73% to 15.46%; removing free initialization yields 10.62% average accuracy, while removing imprinting lowers average incremental accuracy to 62.63%.Without free initialization, accuracy falls from 82.57% in the base session to 1.88% in Session 2 and 1.56% in the final session. Without imprinting, final-session accuracy declines from 57.90% to 48.24%; the paper links imprinting’s value to classifier initialization from five examples per novel class.Source: §3.4 · p. 8

4 RELATED WORK

FSCIL combines scarce supervision with class-incremental learning, requiring stability, plasticity, and efficiency. Prior approaches address forgetting through replay, structured objectives, parameter isolation, or frozen backbones, each with trade-offs.

4 RELATED WORK

  • FSCIL combines few-shot supervision with class-incremental learning, where task identity is unavailable at test time and models risk forgetting earlier classes.Novel classes have only a handful of labeled examples, increasing overfitting and interference risks.Source: §4 · p. 8
  • Replay, meta-learning, and structured objectives can improve forgetting or adaptation, but often add training stages, tuning, or replay costs.Examples include meta-learning Chi et al. (2022); Zhou et al. (2022a), pseudo-features Cheraghian et al. (2021); Liu et al. (2022), and prototype-alignment methods Tao et al. (2020); Hersche et al. (2022); Zhang et al. (2021); Zhu et al. (2021).Source: §4 · p. 9
  • Freezing most or all of the backbone reduces training cost and forgetting, but assumes base-session features suffice for future tasks, an assumption that breaks under cross-domain FSCIL.This strategy is described for Zhou et al. (2022a); Yang et al. (2023).Source: §4 · p. 9
  • Parameter-isolation methods for FSCIL, including sparse or winning-ticket subnetworks, can require mask optimization or per-task search and may fragment capacity or limit forward transfer.The cited examples are Mazumder et al. (2021); Kang et al. (2024).Source: §4 · p. 9
  • Prompting on pretrained foundation models relies on external pretraining and departs from the paper’s learning-from-scratch setting, making it a complementary direction rather than its target.The passage also notes limited robustness under domain shift and limited continual representation learning for typically frozen or weakly adapted models.Source: §4 · p. 9

5 CONCLUSION

SBP uses preallocated parameter capacity to balance stability and plasticity in exemplar-free FSCIL. It performs strongly across standard, cross-domain, and long-horizon evaluations while reducing training cost relative to competing methods.

5 CONCLUSION

  • SBP preallocates capacity, freezes parameters associated with past knowledge, and reinitializes future capacity each epoch to preserve stability without implicitly shaping unused parameters.It is combined with prototype-based classification.Source: §5 · pp. 9–10
  • SBP achieves state-of-the-art performance across multiple FSCIL benchmarks while substantially reducing training cost, and performs strongly under cross-domain shifts and an 80-session ImageNet-1K stream.The conclusion cautions that short, distribution-consistent benchmarks alone do not characterize robust behavior over time.Source: §5 · p. 10
  • The results motivate evaluating FSCIL jointly across stability, plasticity, and computational efficiency, including explicit tests of distribution shift and learning horizon.Methods preserving fixed representations can remain competitive when future classes resemble the base distribution, yet degrade when adaptation or long-term interference matters.Source: §5 · p. 10

A EXPERIMENTAL SETUP

Experiments use ResNet-18 and established FSCIL datasets and baselines. For distance-based classification, the method uses only feature dimensions allocated through the current session, with prototypes frozen after introduction.

A EXPERIMENTAL SETUP

  • All models use ResNet-18 and SGD; incremental training fine-tunes assigned parameters for 40 CIFAR-100 or 120 MiniImageNet iterations at learning rate 0.0025.The initial budget slice is 0.85, with approximately 1.8% of parameters per layer allocated to each new task.Source: A EXPERIMENTAL SETUP · pp. 12–13
  • CIFAR-100 and MiniImageNet use 60 supervised base classes followed by eight 5-way 5-shot sessions; CUB-200 uses 100 base classes followed by ten 10-way 5-shot increments.The protocols learn all classes across sequential sessions without revisiting old data.Source: A EXPERIMENTAL SETUP · p. 13
  • The method is compared with reproduced FSCIL baselines under the same experimental settings and backbone for a fair comparison.The baseline set varies by dataset and includes methods such as FACT, NC-FSCIL, OrCo, Comp, and WaRP.Source: A EXPERIMENTAL SETUP · p. 13

A.1 PARTIAL DISTANCE CLASSIFICATION

  • Distance comparisons use only feature dimensions allocated through the current session, excluding unstable future-reserved dimensions as the active classification subspace grows.For convolutional backbones, final convolution output channels correspond to feature dimensions after global pooling.Source: §A.1 · pp. 13–14
  • New-class prototypes are imprinted from support features and then frozen; inference computes distances over dimensions allocated up to the current session.The projection excludes dimensions that remain unallocated at that session.Source: §A.1 · p. 14

B INCREMENTAL TRAINING TIME COMPARISON WITH SOTA

On CIFAR-100, SBP combines fast base-session training with low incremental-update latency and the best reported accuracy among the compared SOTA baselines. Backbone-frozen methods can update slightly faster incrementally but have lower accuracy in relevant settings.

B INCREMENTAL TRAINING TIME COMPARISON WITH SOTA

  • 14 minutes for SBP’s base session is 1.36× faster than WaRP and almost 5× faster than SoftNet; SBP also achieves the best accuracy compared with SOTA baselines.Several competing methods require over one hour of base training, with ALICE and Approx exceeding 100 minutes.Source: B INCREMENTAL TRAINING TIME COMPARISON… · pp. 14–15
  • 0.04–0.06 minutes per incremental session is SBP’s reported runtime, versus 0.03 minutes for backbone-frozen FACT and NC-FSCIL in most sessions.ALICE requires approximately 1 minute per incremental session, while WaRP requires approximately 0.05–0.08 minutes.Source: B INCREMENTAL TRAINING TIME COMPARISON… · pp. 14–15
  • Backbone-frozen methods’ slightly faster incremental updates come with lower accuracy, particularly when later-session data differs from the base-session data.The passage also notes that these methods often require substantially longer base training to reach peak performance.Source: B INCREMENTAL TRAINING TIME COMPARISON… · pp. 14–15

C RESULTS ON CUB-200-2011

On CUB-200-2011, SBP achieves the best accuracy across most sessions and outperforms the strongest competing method, Comp, on both final and average accuracy.

C RESULTS ON CUB-200-2011

  • SBP improves final-session accuracy by 3.40% and average accuracy by 5.66% over Comp on CUB-200-2011.SBP records 62.51% final-session accuracy and 71.28% average accuracy.Source: C RESULTS ON CUB-200-2011 · p. 15
  • SBP achieves the best accuracy across most sessions in the 10-way 5-shot CUB-200-2011 setting.The dataset contains 200 bird species and 11,788 images.Source: C RESULTS ON CUB-200-2011 · p. 15
  • The CUB-200-2011 results extend SBP’s reported performance beyond CIFAR-100 and MiniImageNet.All methods use the same training settings and backbone described in Appendix A.Source: C RESULTS ON CUB-200-2011 · p. 15

D CROSS-DOMAIN MINI-IMAGENET TO CALTECH-101

In a Mini-ImageNet-to-Caltech-101 FSCIL setting, SBP remains strong as it adapts to a target domain from five examples per class. It leads in average and final-session accuracy, although ALICE is ahead in the first two sessions.

D CROSS-DOMAIN MINI-IMAGENET TO CALTECH-101

  • The experiment trains on 60 Mini-ImageNet base classes, then adds 10 Caltech-101 classes per session using five examples per class.The setup tests adaptation across a domain shift from the base-training data.Source: D CROSS-DOMAIN MINI-IMAGENET TO… · pp. 15–16
  • 64.87% average accuracy and 58.77% final-session accuracy make SBP the top method over the Mini-ImageNet-to-Caltech-101 sequence.SBP exceeds ALICE’s 62.64% average accuracy and WaRP’s final-session accuracy by 4.38%.Source: D CROSS-DOMAIN MINI-IMAGENET TO… · p. 16, Table 9 · p. 16
  • SBP overtakes ALICE by Session 3 and leads every later session, with a 5.77% gap by Session 10.ALICE leads SBP at Sessions 1 and 2: 71.60% and 68.90%, compared with SBP’s 70.34% and 67.88%.Source: D CROSS-DOMAIN MINI-IMAGENET TO… · p. 16
  • ADBS falls from 82.10% base-session accuracy to 26.60% by Session 10, while SBP retains 58.77% after the incremental sequence.SBP starts at 77.33%; the comparison highlights that high base accuracy alone is not sufficient for robust cross-domain FSCIL.Source: D CROSS-DOMAIN MINI-IMAGENET TO… · p. 16
  • SBP’s strong results extend to incremental classes drawn from a different dataset and domain than the base classes.WaRP also remains comparatively stable after the shift, but SBP maintains higher accuracy throughout the incremental sequence.Source: D CROSS-DOMAIN MINI-IMAGENET TO… · p. 16

E LONG-HORIZON IMPLEMENTATION DETAILS AND VIT RESULTS

On an 80-session ImageNet-1K stream, SBP retains accuracy across a long learning horizon, with outcomes depending on backbone. The experiment uses 600 base classes and introduces the remaining 400 over 80 incremental sessions.

E LONG-HORIZON IMPLEMENTATION DETAILS AND VIT RESULTS

  • The ImageNet-1K stream uses 600 base classes and distributes the remaining 400 classes across 80 incremental sessions under a 5-way 5-shot protocol.The ResNet-18 backbone is trained from scratch on the base classes; each incremental session uses 75 training iterations, with 90% of capacity allocated to the base session and 10% distributed across incremental sessions.Source: E LONG-HORIZON IMPLEMENTATION DETAILS… · pp. 16–17
  • 39.08% accuracy is retained by SBP (ResNet-18) after 80 incremental sessions, while WaRP falls from 51.40% at Session 30 to 12.90% at Session 80.WaRP slightly outperforms SBP at Sessions 20 and 30, but later degrades.Source: E LONG-HORIZON IMPLEMENTATION DETAILS… · p. 17, Table 10 · p. 17
  • 45.07% accuracy is retained by SBP (ViT-B/16) at Session 80, compared with 39.08% for SBP (ResNet-18).This is an architecture-level robustness check, not a direct comparison with the ResNet-18 baselines.Source: E LONG-HORIZON IMPLEMENTATION DETAILS… · p. 17

F ARCHITECTURE STUDY

SBP’s accuracy improves with larger tested backbones, while base-session capacity budgets reveal a strong gain from 50% to 75% and diminishing returns near full allocation. Epoch-wise free-parameter reinitialization is intended to decouple current learning from capacity reserved for future sessions.

F ARCHITECTURE STUDY

  • Across tested backbones, SBP’s average accuracy rises from 51.30% with CNNx4 to 72.20% with ResNeXt-29, with ResNet-50 at 70.68%.The study reports gradual accuracy decline across sessions rather than abrupt collapse across the tested architecture families.Source: F ARCHITECTURE STUDY · p. 17

G EFFECT OF BASE SESSION BUDGETS

  • On the 8-session CIFAR-100 budget study, increasing the base allocation from 50% to 75% raises average accuracy from 62.54% to 67.22% and final-session accuracy from 52.14% to 55.87%.The training schedule and incremental-session hyperparameters are held fixed while the base-session capacity budget varies.Source: G EFFECT OF BASE SESSION BUDGETS · pp. 17–18
  • 95% base capacity yields the best reported average accuracy, 69.60%, while 99% yields 69.03%, indicating diminishing returns near full allocation.The authors suggest that near-full base allocation leaves incremental sessions with limited capacity to reshape global features.Source: G EFFECT OF BASE SESSION BUDGETS · p. 18
  • Moderate base budgets of 75%–90% recover most attainable performance while preserving lower base-training cost, with 75% offered as a compromise when base training must be quick.The reported incremental behavior remains stable under this reduced base budget.Source: G EFFECT OF BASE SESSION BUDGETS · p. 18

H THEORETICAL MOTIVATION FOR FREE NEURON RE-INITIALIZATION IN SBP

  • If free parameters remain static, current-task weights can co-adapt to them, so later reassignment can inherit parameters biased by earlier training.The representation is described as a combination of past, current, and future parameter subspaces.Source: H THEORETICAL MOTIVATION FOR FREE… · p. 18
  • Epoch-wise free-parameter resets are intended to make current-task representations invariant to the free subspace and preserve future parameters independently of prior optimization.The authors report that removing reinitialization causes base-class feature clusters to collapse after Session 2, whereas reinitialization keeps them compact and separated.Source: H THEORETICAL MOTIVATION FOR FREE… · pp. 19–20, Figure 2 · p. 19

I FEATURE-SPACE ANALYSIS OF FREE-PARAMETER REINITIALIZATION

Removing epoch-wise re-initialization of free parameters causes abrupt model failure and feature-space collapse. With re-initialization, base representations remain untouched when those parameters are assigned and updated in Session 2.

  • Without free-parameter re-initialization, accuracy plummets to ∼1.56% by the final session.The ablation reports this as abrupt model failure.Source: I FEATURE-SPACE ANALYSIS OF… · p. 20
  • Without re-initialization, five CIFAR-100 base-class clusters that were distinct after Session 1 scatter and overlap after Session 2.The feature embeddings are visualized with t-SNE.Source: I FEATURE-SPACE ANALYSIS OF… · p. 20
  • With epoch-wise re-initialization, free parameters remain non-stationary during the base session, so Session 2 updates leave base representations untouched.This prevents latent co-adaptation and preserves capacity for future sessions without explicit forward masking at inference.Source: I FEATURE-SPACE ANALYSIS OF… · p. 20

Related papers

Loading 2610.04003v1…