Source-linked AI summary
SPECTRA: Subspace-Preserving Embedding Calibration, Transport, and Replay for Fully Few-Shot Class-Incremental Audio Classification
Giries Abu Ayoub, Loay Mualem, Simon Korman
TL;DR
FFCAC must learn new audio classes from a few examples per session without an abundant base dataset or forgetting prior classes. SPECTRA adapts frozen audio–language embeddings, replays old classes through low-rank feature subspaces, and refines prototypes with transductive optimal transport. Across three benchmarks, it improves average accuracy and reduces forgetting over TAPE, with ablations supporting subspace-structured replay.
Problem
FFCAC requires learning incrementally arriving audio classes from few examples while preserving previous classes without an abundant base session.
Method
SPECTRA combines a trainable adapter, exemplar-free low-rank subspace feature replay, and transductive optimal-transport prototype refinement on a frozen encoder.
Results
SPECTRA improves average accuracy and reduces forgetting over TAPE across NSynth-100, FSC-89, and LS-100.
Takeaways & Limitations
Ablations support that replaying subspace structure, rather than replay alone, drives the gain and diminishes forgetting.
Abstract
from arXiv · showhide
Fully few-shot class-incremental audio classification (FFCAC) requires recognizing new sound classes from only a handful of labeled examples per session, without forgetting previously learned classes and without any large base dataset. Existing methods typically freeze a pre-trained audio--language encoder and classify with point prototypes, but they suffer from significant performance degradation throughout the sessions due to generic feature representations. We propose SPECTRA, a framework built on a frozen encoder which adds three components. (i) a lightweight trainable adapter that calibrates the generic embeddings to the task; (ii) subspace feature replay, an exemplar-free anti-forgetting scheme that replays old classes by sampling from the low-rank subspace of their stored features; and (iii) a transductive optimal-transport refinement of prototypes at test time. Our central finding is that the subspace structure of the replay diminishes forgetting and outperforms naive Gaussian replay of equal variance. On three FFCAC benchmarks (NSynth-100, FSC-89, LS-100), SPECTRA improves average accuracy and reduces forgetting over current state-of-the-art methods, and our ablations statistically validate each component.
1. INTRODUCTION
FFCAC requires learning new audio classes from only a few examples while retaining earlier classes, without an abundant base session. SPECTRA addresses limitations of frozen audio–language representations by adding task calibration, exemplar-free replay, and transductive prototype refinement.
- Problem: FFCAC combines few-shot overfitting and incremental catastrophic forgetting because every session, including the first, has only N classes with K examples.The setting removes the assumption of an abundant base session.
- Prior work: TAPE freezes an audio–language encoder and uses a closed-form task transform plus prototype evolution, establishing the current FFCAC state of the art.Its prototype evolution refines class prototypes using low-entropy query samples.
- Motivation: Generic frozen representations can miss task-specific structure, while few-shot audio classifiers can latch onto spurious correlations.These limitations motivate task-specific representation calibration.
- SPECTRA: SPECTRA combines exemplar-free, subspace-structured feature replay with transductive transport in TAPE’s frozen audio–language pipeline.Replay synthesizes old-class features without storing exemplars, and transport uses joint query statistics through Sinkhorn optimal transport.
2. METHODOLOGY
SPECTRA keeps the audio–language encoder frozen while adapting embeddings, replaying prior classes through feature subspaces, and refining prototypes with transductive transport. The pipeline targets continual recognition under five-shot, exemplar-free sessions.
- Problem setup: FFCAC presents five-shot classes incrementally while previous-session samples become unavailable, requiring new-class learning without losing earlier knowledge.The encoder maps clips to d-dimensional embeddings, with d = 1024.
- Pipeline: SPECTRA applies a frozen audio–language encoder, a trainable residual-MLP adapter, TAPE’s orthogonalising transform, and a prototypical classifier.The adapter calibrates embeddings while the transform maps features and prototypes into an orthogonal metric space.
- Prototype refinement: Transductive optimal transport refines prototypes during inference using the unlabeled query batch.The refinement jointly considers queries rather than classifying each sample independently.
- Subspace feature replay: Subspace feature replay represents each class with a compact low-dimensional subspace and synthesizes representative embeddings for continual rehearsal without retaining raw audio.Stored support features provide the class representation used for replay.
W1 LN(x)
SPECTRA adapts frozen audio–language embeddings, rehearses prior classes through low-rank subspaces, and refines prototypes using unlabeled test batches. Its replay preserves class-specific variation rather than isotropic variation, supporting continual learning without retaining raw audio.
- W1 LN(x): A lightweight adapter calibrates frozen audio–language features for the target task and is applied to support and query embeddings before classification.The encoder remains frozen, while the adapter is optimized with the classifier.
- 2.4. Subspace Feature Replay: Subspace feature replay represents each class with a prototype and compact low-dimensional subspace, then synthesizes embeddings for rehearsal without storing raw audio.The subspace is estimated from principal directions of centered support embeddings.
- Training: Training combines current-session cross-entropy with replay loss over synthesized embeddings to regularize the evolving representation.The synthetic embeddings pass through the current adapter and classifier like real samples.
- 2.4. Subspace Feature Replay: Unlike isotropic Gaussian replay, subspace replay preserves each class’s principal variation directions and produces more representative rehearsal samples.Figure 2 reports that subspace replay remains class-aligned across sessions, whereas Gaussian replay produces overlapping clouds.
- 2.5. Transductive Prototype Refinement: Entropy-regularized optimal transport jointly refines prototypes from the unlabeled test batch before final prediction.Sinkhorn iterations produce a soft transport plan, and prototypes are updated using support and softly assigned query embeddings.
3. EXPERIMENTS
SPECTRA improves average accuracy and reduces forgetting across three FFCAC benchmarks, with gains persisting under harder protocols. Ablations identify adapter calibration and structured subspace replay as central contributors, while optimal transport adds accuracy with a small forgetting trade-off.
- Main results: SPECTRA improves average accuracy on NSynth-100, FSC-89, and LS-100 while reducing performance drop relative to FFCAC baselines.The evaluation uses AA and PD under the standard five-session, five-way, five-shot protocol, with means over 50 seeds.
- Per-session analysis: SPECTRA leads at every session, with its NSynth-100 margin over TAPE increasing from +1.5 at S0 to +3.9 at S4.The largest reported average-accuracy effect is +7.1 AA on LS-100.
- Scaling to a harder protocol: SPECTRA still beats TAPE by a wide margin under the harder 10-way, 10-session protocol across the evaluated datasets.FSC-89 uses eight sessions because it contains 89 classes.
- Ablations: Removing the adapter causes the largest single AA drop, indicating that calibrating frozen embeddings is the main driver among the ablated components.The adapter calibrates representations without fine-tuning the frozen encoder.
- Ablations: Subspace replay beats matched-variance isotropic Gaussian replay on two datasets and both metrics, isolating replay structure as the anti-forgetting source.On NSynth-100, Gaussian replay barely improves over no replay.
- Ablations: Optimal transport raises AA by +0.3/+0.5 across the chain but increases PD by +0.4/+0.9, while subspace replay more than compensates.The framework retains OT for its accuracy gain and relies on replay for anti-forgetting.
4. CONCLUSION
SPECTRA combines calibrated frozen embeddings, exemplar-free subspace replay, and transductive prototype refinement for FFCAC. Across three benchmarks, it improves average accuracy and reduces forgetting versus TAPE, with ablations attributing gains specifically to replay subspace structure.
- SPECTRA combines a trainable adapter, exemplar-free low-rank subspace feature replay, and transductive optimal-transport prototype refinement.The adapter calibrates frozen audio–language embeddings, replay samples old classes from stored feature subspaces, and transport refines prototypes at test time.
- Across three FFCAC benchmarks, SPECTRA improves average accuracy and reduces forgetting over TAPE, the current state of the art.
- Ablations statistically establish that replay subspace structure—not replay alone—drives the gain.
- Future work will couple the transport plan with replay subspaces to further stabilize noisier datasets.