PaperlayerView exact revision on arXiv

Source-linked AI summary

Investigating catastrophic forgetting in sound event classification

Riccardo Casciotti, Annamaria Mesaros

arXiv:2609.11447v1eess.AScs.AIcs.SD

TL;DR

This paper examines how to prevent catastrophic forgetting when learning sound-event classes incrementally. It compares architectural and regularization approaches, including a dynamic classifier head and methods that protect network kernels, on FSD50K and AudioSet. Fine-tuning, which freezes the feature extractor and updates the dynamic head, shows little forgetting and stable training in the studied setting.

  • Problem

    Deep neural networks can forget previously learned knowledge when trained on new information, making it challenging to learn tasks incrementally while retaining earlier knowledge.

    Source: §I · p. 1

  • Method

    The study compares freezing, regularization, architectural expansion, and kernel-level plasticity modulation for class-incremental sound event classification.

    Source: §I · p. 2, A. Architectural methods · pp. 2–3

  • Results

    Fine-tuning has near-zero backward transfer values on AudioSet and the lowest forgetting among continual-learning methods on FSD50K, while rank-based and learning-modulation methods show more instability.

    Source: §V · p. 5

  • Takeaways & Limitations

    The results suggest that, in this setting, organizing outputs for new classes matters more than heavily modifying or constraining the feature extractor.

    Source: §V · p. 6

Abstract

from arXiv · show

This work investigates a number of approaches to prevent catastrophic forgetting in class incremental learning scenarios for sound event classification tasks. We analyze the problem using architectural and regularization approaches, using FSD50K and AudioSet datasets. We design incremental stages and solutions that selectively protect the kernels of the network from weight updates to prevent catastrophic forgetting, and a dynamic head solution that expands itself each time a new task is learned. The findings show that catastrophic forgetting mainly happens in deeper layers, in particular in the classifier head. For the studied in-domain sound classification problem, the solution that seems to alleviate catastrophic forgetting and is the most efficient is a full freezing of the feature extractor with a fine-tuning of the dynamic head classifier, showing little to no forgetting and great training stability, and a good balance between memory-stability and learning plasticity.

I. INTRODUCTION

Class-incremental learning asks neural networks to acquire new tasks while retaining earlier knowledge, a stability–plasticity challenge made harder by complex, multi-label sound events. This paper compares several strategies for that challenge in sound event classification.

I. INTRODUCTION

  • Class-incremental learning introduces new class groups over tasks and evaluates the model on all classes seen so far, requiring a balance between learning new classes and retaining prior knowledge.Multi-label audio clips can contain co-occurring events, complicating the preservation of representations across incremental steps.Source: §I · p. 1
  • Continual-learning strategies include loss-based regularization and methods that directly modify parameter or kernel updates to protect information from earlier tasks.The introduction also describes using a pruning-inspired criterion to score kernels and freeze selected proportions during incremental learning.Source: §I · pp. 1–2
  • In class-incremental learning, the classifier output space grows as new classes arrive, commonly by adding output units for those classes.The paper uses this classifier expansion mechanism across its methods.Source: §I · p. 2
  • The study compares freezing, regularization, architectural expansion, and kernel-level plasticity modulation for multi-label, class-incremental sound event classification on FSD50K and AudioSet.Experiments include different task configurations and class orderings, with standard audio-classification and continual-learning metrics.Source: §I · p. 2

II. METHODS

The methods span architectural and regularization strategies, with interventions focused on individual kernel weights and the classifier head.

II. METHODS

  • The study implements continual-learning approaches ranging from architectural to regularization solutions, focusing on individual kernel weights and the final fully connected layer, termed the classifier head.Source: §II · p. 2

A. Architectural methods

The architectural methods expand the classifier head as classes arrive and use several approaches to protect earlier knowledge, from distillation and kernel-level controls to freezing the feature extractor.

A. Architectural methods

  • The dynamic classifier head expands its outputs as tasks arrive, assigning one output to each class while keeping outputs for earlier tasks available.Source: A. Architectural methods · p. 2
  • Knowledge distillation adds a loss that encourages the model on a new task to preserve the previous model’s behavior on earlier classes.The paper uses the distillation loss, rather than the cosine-similarity loss in the referenced approach.Source: A. Architectural methods · pp. 2–3
  • Kernel protection ranks kernels by task relevance and prevents selected high-ranked kernels from receiving updates during later-task training.The method uses a passive filter-pruning-inspired ranking and tests protection proportions of 12.5%, 25%, 50%, 75%, and 100%.Source: A. Architectural methods · p. 3
  • Learning-modulation ranks kernels by task activation and adjusts later updates using stored activation and average-update statistics.It reduces plasticity for important kernels when updates exceed their previously observed average, while allowing less-critical kernels increased plasticity.Source: A. Architectural methods · p. 3
  • Fine-tuning freezes all convolutional and batch-normalization layers after the first task, leaving only the classification head to update on later tasks.Source: A. Architectural methods · p. 3

III. DATA AND EVALUATION

The experiments use AudioSet and FSD50K, two large-scale datasets for sound-event classification.

III. DATA AND EVALUATION

  • The experiments use AudioSet, with approximately 2.1 million 10-second segments and 527 event classes, and FSD50K, with 51,197 clips and 200 classes.AudioSet segments have one or more labels; FSD50K clips are annotated with classes derived from the AudioSet ontology.Source: §III · p. 3

A. Data preprocessing and experimental setup

The study uses the 50 largest classes from each dataset and converts audio to fixed-length log-mel spectrogram inputs.

B. Evaluation metrics

Performance is measured with mAP, alongside continual-learning metrics that track forgetting, interference on earlier tasks, and the gap from joint training.

  • Mean Average Precision (mAP) is the main performance metric and is suited to multi-label classification.Source: B. Evaluation metrics · p. 4
  • Forgetting Measure (FM) directly quantifies forgetting on previous tasks by averaging their maximum observed mAP declines.Source: B. Evaluation metrics · p. 4
  • Backward Transfer (BWT) measures interference from new tasks on previously learned tasks.Source: B. Evaluation metrics · p. 4
  • Intransigence Measure (IM) compares incremental-learning performance with a jointly trained reference model.The reference mAP is for a model trained jointly on task k; mAP_k,k is performance immediately after training on task k.Source: B. Evaluation metrics · p. 4

IV. EXPERIMENTAL SETUP

The experiments simulate five sequential class-incremental tasks, evaluating performance on each task and retention of previously learned information.

  • Each experiment trains across five tasks, then evaluates performance on each task and retained information from earlier training.Source: §IV · p. 4
  • The first task contains 30 classes and each of the four later tasks contains five disjoint classes, with old classes absent from later training.Source: §IV · p. 4
  • Both datasets use ordered and not ordered class configurations, with the latter averaged over five random class assignments.Ordered learning places the highest-sample classes first; not ordered allows high-sample classes to appear later.Source: §IV · p. 4

A. Model Architecture

The model uses a PANNs-style convolutional network with a dynamic fully connected classifier head for multi-label sound classification.

B. Training and validation

Training proceeds through disjoint class subsets, expanding the output head as new tasks arrive; each task is validated using only its own validation data.

B. Training and validation

V. RESULTS AND DISCUSSION

Across the studied setups, fine-tuning is particularly stable, while rank-based strategies show greater instability. The findings point to classifier-level interference and output-space organization, rather than major feature-extractor changes, as central to performance.

V. RESULTS AND DISCUSSION

  • Fine-tuning shows moderate degradation across tasks and consistently outperforms rank-based modulation strategies.The evaluation reports cumulative mAP over all classes learned so far across the task sequence.Source: §V · pp. 4–5, Figure 2 · p. 5
  • On AudioSet, fine-tuning has BWT values close to zero, while rank-based strategies and learning-modulation show more negative BWT and higher FM and IM.FSD50K shows the same pattern: fine-tuning has the lowest forgetting among continual-learning methods, while KLD and other approaches show larger degradation.Source: §V · p. 5, Table I · p. 5, Table II · p. 5
  • More restrictive rank-based configurations generally perform better, while looser constraints tend to be less stable.Variants leaving a larger portion of the network adaptable usually have worse BWT and IM values than more aggressive constraints.Source: §V · p. 5
  • The results suggest that the main bottleneck is output-space organization rather than feature-extractor evolution.A shared output structure performs strongly, while feature-level modulation offers limited benefit; suitable outputs for new classes may reduce classifier-level interference.Source: §V · p. 6
  • The experiments indicate that complex feature-level continual-learning mechanisms are not clearly advantageous in this scenario.The feature extractor appears to retain useful general acoustic representations across the incremental sequence, while degradation is more likely related to classifier-level interference and output structure.Source: §V · p. 6
  • For FSD50K, ordered learning is more beneficial than unordered learning across the continual-learning metrics.One proposed explanation is that most examples are single-labeled, making the largest initial 30-class task the best available representation for future tasks.Source: §V · p. 6

VI. CONCLUSIONS

The conclusions identify the dynamic head as the most helpful architectural approach and locate most forgetting in the classifier head. Feature reuse without modulation is supported for tasks sharing a probability distribution.

VI. CONCLUSIONS

  • Among the architectural approaches, the dynamic head seems to help most with catastrophic forgetting.The authors also report that regularization approaches negatively affect both forgetting and convergence.Source: §VI · p. 6
  • The findings suggest catastrophic forgetting happens mainly in the classifier head.The authors assume past-task features can be used by later tasks without modulation when tasks follow the same probability distribution.Source: §VI · p. 6
  • Regularization approaches negatively affect both forgetting and network convergence.Source: §VI · p. 6
  • Future work will investigate differing class distributions and develop effective approaches to alleviate catastrophic forgetting.Source: §VI · p. 6
Loading 2609.11447v1…