PaperlayerView exact revision on arXiv

Source-linked AI summary

Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima

Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, Xiao-Ming Wu

arXiv:2111.01549v2cs.LGcs.CV

TL;DR

Incremental few-shot learning must add new classes from scarce data without forgetting previously learned classes, but data scarcity intensifies catastrophic forgetting. The paper addresses this at the base-training stage by finding flat local minima and fine-tuning within the resulting flat region, effectively mitigating forgetting and adapting to new classes.

  • Problem

    Incremental few-shot learning aims to continually learn new tasks from only a few examples, but scarce data makes catastrophic forgetting more severe as models overfit new classes and forget old ones.

  • Method

    The method searches for flat local minima during base-class training, then fine-tunes parameters within the flat region during incremental few-shot learning.

  • Results

    Extensive benchmark experiments show that the approach effectively mitigates catastrophic forgetting and adapts to new classes, outperforming prior state-of-the-art methods and approaching the approximate upper bound.

  • Takeaways & Limitations

    Addressing forgetting during base training enables new-class adaptation while preserving previously learned classes in incremental few-shot learning.

  • Takeaways & Limitations

    The convergence analysis relies on a strict unbiased-gradient assumption that can be relaxed to a broader condition, and assumes bounded gradient variance.

Abstract

from arXiv · show

This paper considers incremental few-shot learning, which requires a model to continually recognize new categories with only a few examples provided. Our study shows that existing methods severely suffer from catastrophic forgetting, a well-known problem in incremental learning, which is aggravated due to data scarcity and imbalance in the few-shot setting. Our analysis further suggests that to prevent catastrophic forgetting, actions need to be taken in the primitive stage -- the training of base classes instead of later few-shot learning sessions. Therefore, we propose to search for flat local minima of the base training objective function and then fine-tune the model parameters within the flat region on new tasks. In this way, the model can efficiently learn new classes while preserving the old ones. Comprehensive experimental results demonstrate that our approach outperforms all prior state-of-the-art methods and is very close to the approximate upper bound. The source code is available at https://github.com/moukamisama/F2M.

1 Introduction

Incremental few-shot learning must add new categories from scarce data while retaining prior knowledge, but scarcity and imbalance intensify catastrophic forgetting. The paper therefore shifts prevention to base training by finding flat minima and fine-tuning within their regions.

  • Incremental few-shot learning continually learns new tasks from only a few examples, unlike ordinary incremental learning with sufficient new-class data.
  • Small new-task datasets cause models to overfit new classes and rapidly forget old ones, producing catastrophic performance drops.
  • A base-only intransigent model consistently outperforms state-of-the-art methods, including joint training affected by severe data imbalance.
  • The proposed method searches for flat local minima during base training, then fine-tunes parameters within the flat region during subsequent few-shot tasks.Random parameter noise and joint optimization find the region; parameter clamping keeps updates inside it.
  • Experiments on CIFAR-100, miniImageNet, and CUB-200-2011 outperform all state-of-the-art methods and approach the approximate upper bound.

2 Related Work

Related work addresses incremental few-shot learning through parameter or exemplar constraints, while this paper connects flat-minima optimization to mitigating forgetting during later adaptation.

  • Few-shot learning methods primarily use optimization-based adaptation or metric-based similarity in learned embedding spaces.
  • Incremental learning seeks continual recognition of new classes without forgetting old classes, often constraining parameter changes or preserving exemplars.
  • Incremental few-shot methods include topology preservation, selective parameter updates, exemplar embedding constraints, and fixed embedding networks.
  • Flat local minima have been associated with better generalization and robustness to shifted test loss under random perturbations.
  • This paper applies flat-minima search in the base session and tunes within the resulting region to mitigate catastrophic forgetting.

3 Severity of Catastrophic Forgetting in Incremental Few-Shot Learning

Incremental few-shot learning is especially vulnerable to forgetting because later sessions provide little data and may expose the model to only current classes. A base-only baseline nevertheless outperforms adapted methods, indicating that preserving old classes is critical.

  • Problem Statement: IFL uses a data-rich base session followed by small N-way K-shot sessions, with access limited to current data and possibly saved exemplars.
  • Problem Statement: Catastrophic forgetting is the drastic drop in previous-class test performance caused by unavailable old data, and data scarcity makes it worse in IFL.
  • A Simple Baseline Model for IFL: The intransigent baseline trains only on the base session, fixes its feature extractor thereafter, and uses nearest class mean inference across sessions.
  • A Simple Baseline Model for IFL: In nearest class mean classification, each class prototype is the mean embedding of its training images, and Euclidean distance determines similarity.
  • A Simple Baseline Model for IFL: The base-only baseline outperforms state-of-the-art IFL, IL, and joint-training methods on miniImageNet, CIFAR-100, and CUB-200.
  • A Simple Baseline Model for IFL: For IFL, preserving base classes may matter more than adapting to new classes because few-shot gains cannot offset the larger base-class performance drop.
  • A Simple Baseline Model for IFL: Parameter or exemplar constraints applied during new-class fine-tuning do not effectively prevent catastrophic forgetting, suggesting intervention during base training.

4 Overcoming Catastrophic Forgetting in IFL by Finding Flat Minima

The approach addresses catastrophic forgetting at base training by finding a flat minimum, then fine-tuning within its bounded region during later few-shot sessions. Noisy-loss optimization, parameter clamping, exemplar replay, and metric-based classification support adaptation while preserving base knowledge.

  • Motivation: Flat local minima are sought during base training because preventing forgetting later in few-shot sessions may be too late.The method targets the primitive base session rather than only subsequent incremental sessions.
  • Flat-minimum formulation: A b-flat local minimum keeps the objective low throughout θ⋆−b ⪯ θ ⪯ θ⋆+b and higher outside this region.The definition requires equal loss within the bounded region and larger loss on specified neighboring intervals outside it.
  • Base-stage optimization: Approximate flat regions are found by repeatedly adding bounded random noise and jointly optimizing the resulting loss estimates.The objective minimizes expected loss over data and noise, approximated empirically through multiple noise samples.
  • Incremental adaptation: Metric-based fine-tuning uses Euclidean distance, while few-shot scarcity motivates only a small number of update iterations to limit overfitting.The classifier considers all encountered classes through their prototypes.
  • Incremental adaptation: During sessions t ≥2, the embedding parameters are fine-tuned on current data and saved exemplars, then clamped to φ⋆−b ⪯ φ ⪯ φ⋆+b.The algorithm also saves normalized prototypes for base and new classes and uses the union of current data with prior exemplars.
  • Convergence analysis: The convergence analysis optimizes the noisy expected-loss objective with stochastic gradient descent under smoothness and boundedness assumptions.The theorem establishes convergence of the proposed algorithm, with assumptions limiting gradient variation and the objective’s behavior.

5 Experiments

Experiments evaluate F2M across benchmark datasets, compare it with prior methods, and analyze the flat-minima design and its components. F2M consistently outperforms state-of-the-art methods, while ablations support restricting parameters within a flat region.

  • Comparison with the State-of-the-Art: Experiments compare F2M with state-of-the-art methods on CIFAR-100, miniImageNet, and CUB-200-2011 incremental few-shot tasks.The evaluations use 5-way 5-shot tasks on CIFAR-100 and miniImageNet, and 10-way 5-shot tasks on CUB-200-2011.
  • Comparison with the State-of-the-Art: F2M outperforms state-of-the-art methods and the Baseline, with a 0.2% gap to cRT in the last miniImageNet session.cRT is treated as the approximate upper bound because it addresses data imbalance and exceeds the Baseline by 1% to 3%.
  • Comparison with the State-of-the-Art: The Baseline outperforms existing state-of-the-art approaches on all incremental sessions, indicating severe catastrophic forgetting in those methods.The Baseline only trains on base classes and does not tune on new tasks.
  • Analysis on the flatness of local minima: Table 4 compares the flatness of local minima found by the Baseline and F2M, using an indicator and variance computed from repeated noise injections.The reported values demonstrate that F2M finds a more flat local minimum.
  • Ablation Study and Analysis: Removing prototype fixing causes a performance drop of nearly 0.6%, while removing parameter clamping also causes a significant performance drop.Prototype fixing preserves saved base-class prototypes against semantic drift, and parameter clamping restricts parameters to the b-flat region.
  • Ablation Study and Analysis: The best flat-region results occur for b ∈[0.005, 0.02]; larger bounds reduce base-class performance, while smaller bounds reduce new-class performance.The study evaluates session 1 and session 9 accuracy on CIFAR-100 5-way 5-shot incremental learning.

6 Conclusion

The paper addresses catastrophic forgetting by finding flat local minima during base training and fine-tuning within the resulting region on new tasks. Experiments show effective mitigation of forgetting and adaptation to new classes, but the method may not suit medium- or high-shot tasks because its flat region limits model capacity.

  • F2M finds flat local minima during base training and fine-tunes model parameters within the flat region on new tasks.
  • Extensive benchmark experiments show that F2M can mitigate catastrophic forgetting and adapt to new classes.
  • The method may not suit medium- or high-shot tasks because its relatively small flat region limits model capacity.The paper suggests searching for a less flat but wider region as a possible adaptation for incremental learning.

Checklist

The checklist records affirmative coverage of claims, reproducibility materials, training details, theoretical assumptions and proofs, and limitations. It also marks asset licensing and human-subject documentation items as not applicable where relevant.

  • The paper states that its main claims accurately reflect its contributions and scope and that its limitations are described in Section 6.
  • The theoretical-results checklist reports that assumptions are stated and complete proofs are included.
  • The checklist confirms that code, data, reproduction instructions, training details, and compute resources are documented or linked.
  • Existing assets are cited, while asset licensing, new-asset release, consent, personal-information review, offensive-content review, and human-subject procedures are marked not applicable where indicated.

A.1 Proof of Theorem 4.1

The proof establishes convergence of the expected objective under the stated assumptions and then derives asymptotic vanishing of the expected squared gradient norm.

  • Proof of convergence: Taking total expectations and summing the update inequalities yields a bound involving the objective changes and step-size terms.The proof rearranges the inequality and divides by the cumulative step-size weights.
  • Theorem consequence: The convergence of F(θ_k), together with the preceding lemma, implies that ∥∇R(θ_k)∥_2^2 cannot remain asymptotically far from zero.The theorem strengthens the lemma's consequence under twice differentiability and L2-smoothness.
  • Proof setup: The proof uses assumptions on bounded risk, gradient estimates, step sizes, smoothness, and sampled noise to analyze the parameter updates.The assumptions include a lower risk bound, unbiased-gradient conditions, and smoothness of the risk function.
  • Proof of convergence: The bound is part of a convergent sum because the squared step sizes converge, which makes the accumulated objective-change term bounded.The proof introduces the accumulated positive term A_K^+ and shows that it converges.
  • Proof of convergence: Because the accumulated objective changes converge, the sequence F(θ_k) converges.The proof combines the telescoping relation for F(θ_K) with convergence of A_K^+.

A.2 More Experimental Details

The experiments use dataset-specific image settings and a practical exemplar protocol, while re-implementation checks show results close to prior reports for Rebalance and ICaRL.

  • Dataset settings: CIFAR-100 and miniImageNet use 500 training and 100 test images per class, while CUB-200-2011 uses resized and cropped 224 × 224 images.CIFAR-100 images are 32×32 and miniImageNet images are 84×84.
  • Exemplar setting: The chosen practical setting assumes many base classes with sufficient data and relatively few novel classes lacking training data.The motivation is computational efficiency and efficient storage use.
  • Exemplar setting: ICaRL and Rebalance store 2 exemplars per class for both base and new classes, giving them more exemplars per session than the proposed method.FSLL is re-implemented with the same number of new-class exemplars as the proposed method.
  • Implementation checks: Our re-implementation results for Rebalance and ICaRL are very close to those reported previously.The figure marks the re-implemented results with an asterisk.

A.3 Additional Experiment Results

Additional experiments examine prototype norms, implementation correctness, class-split robustness, uncertainty, and matched TOPIC splits across three datasets. F2M generally remains competitive or superior, with a stated exception on CUB-200-2011.

  • Prototype norms: The average norm of new-class prototypes is significantly smaller than that of old-class prototypes after incremental few-shot learning on CIFAR-100.The experiment uses 60 base classes and 40 new classes.
  • Implementation correctness: Re-implementations of ICaRL, Rebalance, and FSLL achieve results close to those reported in their original papers.The FSLL comparison uses no exemplars for new classes, while ICaRL and Rebalance correctness is evaluated on CIFAR-100.
  • Class-split robustness: F2M consistently outperforms other methods across two CIFAR-100 class splits, including splits with large differences between base and novel classes.The reported robustness concerns class-pattern differences between base and novel categories.
  • Uncertainty: Across ten runs with 95% confidence intervals, F2M shows steady improvement over the state-of-the-art methods included in the comparisons.Each 5-shot run randomly selects 5 samples per class.
  • Matched class splits: On matched TOPIC class splits, F2M and the Baseline consistently outperform other methods on CIFAR-100 and miniImageNet, but joint training performs better on CUB-200-2011.The authors attribute the CUB result partly to less significant data imbalance and differing learning rates during base training.
Loading 2111.01549v2…