Source-linked AI summary
SPECTRA: Subspace-Preserving Embedding Calibration, Transport, and Replay for Fully Few-Shot Class-Incremental Audio Classification
Giries Abu Ayoub, Loay Mualem, Simon Korman
TL;DR
In fully few-shot class-incremental audio classification, a model must learn new sound classes from few examples while retaining earlier classes without revisiting old data. SPECTRA calibrates frozen audio-language embeddings, replays old classes from low-rank feature subspaces, and refines prototypes at inference; across three benchmarks, it improves average accuracy and reduces forgetting over TAPE.
Problem
FFCAC must learn new audio classes from few examples while retaining earlier knowledge without revisiting old data, combining few-shot overfitting and catastrophic-forgetting challenges.
Source: §1 · p. 1, §2.1 · p. 2Method
SPECTRA calibrates frozen audio-language embeddings with a trainable adapter, rehearses prior classes through low-rank feature-subspace replay without raw audio, and refines prototypes via transductive optimal transport.
Source: §2.2 · p. 2, §2.4 · p. 2Results
Takeaways & Limitations
Ablations indicate that the replay's subspace structure, not replay alone, drives the reported gains.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Fully few-shot class-incremental audio classification (FFCAC) requires recognizing new sound classes from only a handful of labeled examples per session, without forgetting previously learned classes and without any large base dataset. Existing methods typically freeze a pre-trained audio--language encoder and classify with point prototypes, but they suffer from significant performance degradation throughout the sessions due to generic feature representations. We propose SPECTRA, a framework built on a frozen encoder which adds three components. (i) a lightweight trainable adapter that calibrates the generic embeddings to the task; (ii) subspace feature replay, an exemplar-free anti-forgetting scheme that replays old classes by sampling from the low-rank subspace of their stored features; and (iii) a transductive optimal-transport refinement of prototypes at test time. Our central finding is that the subspace structure of the replay diminishes forgetting and outperforms naive Gaussian replay of equal variance. On three FFCAC benchmarks (NSynth-100, FSC-89, LS-100), SPECTRA improves average accuracy and reduces forgetting over current state-of-the-art methods, and our ablations statistically validate each component.
1. INTRODUCTION
SPECTRA addresses fully few-shot class-incremental audio classification by calibrating frozen audio–language embeddings, replaying earlier classes through compact subspaces, and refining prototypes with transductive optimal transport. The introduction reports consistent accuracy and forgetting improvements over TAPE across three benchmarks, supported by per-session analyses and significance tests.
Problem
- FFCAC gives every session, including the first, only N classes with K examples each, combining few-shot overfitting with incremental catastrophic forgetting.Unlike FCAC, FFCAC removes the assumption of an abundant base session and requires learning new classes without revisiting old data.Source: §1 · p. 1
Related work
- TAPE freezes an audio–language encoder and combines a closed-form task-adaptive transformation with low-entropy query-based prototype evolution for FFCAC.The introduction identifies TAPE as the first method to exploit an audio–language model for FFCAC and the current state of the art.Source: §1 · p. 1
- SPECTRA extends TAPE’s frozen-encoder pipeline with exemplar-free subspace replay and transductive transport, drawing on prior replay and Sinkhorn-based query-set methods.The introduction also notes that SpurAudio [10] finds few-shot audio classifiers can latch onto spurious correlations, motivating task-specific representation calibration.Source: §1 · p. 1
Motivation
- Because prior-session samples are unavailable while representations must adapt to new classes, SPECTRA uses stored class-feature geometry to synthesize replay embeddings for continual supervision.This compact feature representation is intended to preserve earlier class knowledge without retaining raw audio.Source: §1 · p. 1
Contributions
- Experiments on three FFCAC benchmarks show consistent improvements over TAPE in accuracy and forgetting, supported by per-session analyses and statistical significance tests.The introduction names NSynth-100, FSC-89, and LS-100 as the benchmarks in the paper context.Source: §1 · p. 1
- SPECTRA combines a trainable residual-FFN adapter, compact low-rank subspace feature replay, and transductive optimal-transport refinement of prototypes at inference.The replay synthesizes representative embeddings for continual rehearsal without storing raw audio; the introduction says preserving subspace geometry outperforms naive replay strategies.Source: §1 · p. 1
2. METHODOLOGY
SPECTRA addresses fully few-shot class-incremental audio classification with a frozen encoder, task-specific embedding calibration, subspace feature replay, and transductive prototype refinement.
2.1. Problem Setup
- FFCAC introduces classes incrementally with only K support examples each, while previous sessions become unavailable and the model must preserve earlier knowledge under exemplar-free constraints.The encoder maps audio clips to 1024-dimensional embeddings, and the method retains TAPE’s closed-form Task-Transform unchanged to project prototypes into an orthogonal metric space.Source: §2.1 · pp. 1–2
2.2. Overview
- SPECTRA combines a lightweight embedding adapter, subspace feature replay for continual rehearsal, and transductive optimal-transport prototype refinement at inference.Replay samples synthetic old-class features from each class’s low-rank support-feature subspace; inference uses a Sinkhorn step on the unlabeled query batch.Source: §2.2 · p. 2, Figure 1 · p. 2
2.3. Embedding Adapter
- A lightweight residual adapter calibrates frozen audio-language embeddings to the target taxonomy while avoiding full-backbone fine-tuning, which is prone to overfitting in the fully few-shot regime.The adapter uses a transformer-style feed-forward residual block with LayerScale.Source: §2.3 · p. 2
W1 LN(x)
SPECTRA calibrates frozen-encoder embeddings with a trainable adapter, rehearses prior classes through subspace feature replay, and refines prototypes at test time using transductive optimal transport.
W1 LN(x)
- The adapter calibrates support and query embeddings before the Task-Transform and is the framework’s only substantial trainable component, optimized end-to-end with the classifier.This places all features in the same calibrated embedding space.Source: W1 LN(x) · p. 2
2.4. Subspace Feature Replay
- Low-rank subspace replay synthesizes prior-class embeddings along principal variation directions, preserving class-specific structure without storing raw audio.The synthesized embeddings are processed by the current adapter and classifier like real samples.Source: §2.4 · pp. 2–3
2.5. Transductive Prototype Refinement
- Entropy-regularized optimal transport uses the unlabeled query batch to refine prototypes jointly over T iterations before prediction.Each prototype is updated using its support embeddings and softly assigned query embeddings.Source: §2.5 · p. 3
2.6. SPECTRA
- Training freezes the encoder and closed-form transform while optimizing the adapter, prototypes, and anchors on new-class and replay features; inference applies transport refinement without gradients or parameter updates.The training objective classifies new-class support features and synthetic replay features by cross-entropy.Source: §2.6 · p. 3
3. EXPERIMENTS
SPECTRA improves average accuracy and reduces forgetting across the standard FFCAC benchmarks, and it retains an advantage over TAPE under a harder protocol. Ablations attribute the gains to adapter calibration and low-rank subspace replay, with optimal transport adding accuracy at a small forgetting cost.
Setup
- The standard evaluation uses a frozen PENGI ALM and otherwise retains TAPE’s closed-form transform and training pipeline as the baseline.Unless stated otherwise, settings are adapter expansion r=3, replay rank k=3, replay weight λ=1, and T=3 OT prototype-refinement iterations.Source: §3 · p. 3
Main results
- Against various FFCAC baselines, SPECTRA improves average accuracy on all datasets and reduces forgetting (PD); its largest reported effect is on LS-100, at +7.1 AA.Source: §3 · pp. 3–4
Per-session analysis
- On NSynth-100, SPECTRA’s margin over TAPE grows from +1.5 at S0 to +3.9 at S4, and it leads at every session.The per-session analysis reports that the gains grow over sessions; the supplied passage does not specify the metric for these margins.Source: §3 · pp. 3–4
Scaling to a harder protocol
- Under the harder 10-way protocol, SPECTRA still beats TAPE by a wide margin across the datasets.The setting uses 10 sessions for NSynth-100 and LS-100 and 8 for FSC-89, which has only 89 classes; Table 3 reports per-session accuracy from 30 seeds.Source: §3 · p. 4, Table 3 · p. 4
Ablations: component contribution
- Subspace replay yields the best AA and lowest PD, while removing the adapter causes the largest single AA drop and lowers PD.The component ablation decomposes SPECTRA’s average accuracy and performance drop; no specific cell values are provided in the passages.Source: §3 · p. 4, Table 5 · p. 4
Ablations: replay structure
- Subspace replay beats matched-variance isotropic Gaussian replay on two datasets and both metrics; on NSynth-100, Gaussian replay barely improves over no replay.This comparison indicates that the anti-forgetting effect depends on replay’s anisotropic, low-rank structure, not merely replaying samples.Source: §3 · p. 4
Ablations: adapter capacity
- Sweeping adapter expansion r across {2, 3, 4, 8} changes AA by less than the seed standard deviation on both datasets, so r=3 is used.The paper selects r=3 as the smallest value on the observed plateau.Source: §3 · p. 4
Ablations: transport assignment
- Optimal transport raises AA by +0.3/+0.5 but increases PD by +0.4/+0.9 across the chain; subspace replay more than compensates, producing the best PD overall.The method retains OT for its accuracy gain while relying on replay for anti-forgetting.Source: §3 · p. 4
4. CONCLUSION
SPECTRA combines embedding calibration, exemplar-free subspace feature replay, and transductive optimal-transport refinement for fully few-shot class-incremental audio classification. Across three benchmarks, it improves average accuracy and reduces forgetting over TAPE, with ablations attributing the replay gain to its subspace structure.
4. CONCLUSION
- Across three FFCAC benchmarks, SPECTRA improves average accuracy and reduces forgetting over TAPE, the current state of the art.Ablations statistically establish that the replay’s subspace structure—not replay alone—drives the gain.Source: §4 · p. 4
- SPECTRA combines a trainable adapter for frozen audio–language embeddings, exemplar-free replay from old classes’ low-rank feature subspaces, and transductive optimal-transport refinement.The subspace replay is designed to fight forgetting by sampling from stored feature subspaces.Source: §4 · p. 4
- Future work will couple the transport plan with replay subspaces to further stabilize performance on noisier datasets.Source: §4 · p. 4