PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain-Incremental Learning for Multi-Channel Replay Speech Detection

Michael Neri

arXiv:2609.11194v1eess.AScs.CRcs.SDeess.SP

TL;DR

Replay detectors must learn changing acoustic environments without retaining past speech, but prior continual-learning evidence does not cover multi-channel physical-access replay across such environments. This paper frames the problem as domain-incremental learning, benchmarks standard strategies and TSB across environment orderings, and finds that TSB improves final and incremental accuracy while sequence order strongly shapes final performance.

  • Problem

    The paper addresses domain-incremental multi-channel replay detection, where acoustic environments change over time and past recordings should not be revisited.

  • Method

    The study benchmarks a beamformer-based detector with EWC, GPM, and TSB, which keeps one environment-specific spatial front-end with a shared classifier.

  • Results

    TSB improves final and incremental accuracy, while EWC halves forgetting but loses plasticity, GPM matches naive fine-tuning, and the final environment dominates final EER.

  • Takeaways & Limitations

    Spatial specialization is the supported source of TSB’s gain, while the environment ordering—especially its final environment—strongly influences performance.

  • Takeaways & Limitations

    The experiments retain only the D2 four-microphone linear array, leaving other array geometries for future work.

Abstract

from arXiv · show

Replay attacks are the most accessible threat to voice-controlled systems, and the acoustic cues that expose them are strongly modulated by the environment in which the attack is mounted. A detector deployed in the field therefore has to absorb new acoustic conditions over time, ideally without revisiting past recordings, since retaining speech indefinitely is both expensive and legally constrained. We frame this as Domain-Incremental Learning (DIL) over acoustic environments and present the first continual learning benchmark for multi-channel replay speech detection, evaluating a state-of-the-art beamformer-based detector over all 24 environment orderings of the ReMASC corpus with five seeds. Sequential fine-tuning forgets severely, raising the error rate on previously learned environments by 18.8 points. Elastic weight consolidation (EWC) halves forgetting but loses plasticity, gradient projection memory (GPM) is statistically indistinguishable from naive fine-tuning, and the proposed task-specific beamformer (TSB) that keeps one spatial front-end per environment significantly improves final and incremental accuracy. We further show that the last environment of the sequence dominates final performance. Code, results, and analysis are available at https://github.com/michaelneri/replay-speech-continual.

1. INTRODUCTION

Replay detection must adapt to changing acoustic environments without retaining past speech, motivating domain-incremental learning for multi-channel replay attacks. The paper introduces TSB and benchmarks it alongside continual-learning baselines across environment orderings.

  • Motivation: Replay cues vary with room and recording conditions, so detectors degrade when deployment environments differ from training.Spatial differences include inter-channel timing and levels, coherence, and direct-to-reverberant ratio.
  • Problem formulation: Domain-incremental learning updates a detector on successive acoustic environments while retaining earlier environments without revisiting past recordings.The label space remains fixed while the input distribution changes.
  • Contribution: The paper introduces the task-specific beamformer, which uses one learnable spatial front-end per environment with a shared classifier and requires no environment label at inference.Earlier heads are frozen during later adaptation, and trained-head logits are averaged before softmax.
  • Contribution: The study benchmarks a multi-channel detector with TSB, EWC, and GPM over all environment orderings and five seeds.The benchmark evaluates continual learning under changing acoustic environments rather than changing spoofing labels.

2. METHOD

The method formulates replay detection as domain-incremental binary classification and combines a beamformer-based detector with continual-learning strategies. TSB separates environment-specific spatial processing from a shared classifier while avoiding storage of past utterances.

  • DIL for Replay Speech Detection: Replay detection is binary classification of multi-channel recordings as genuine or loudspeaker-replayed, learned from one acoustic environment at a time.Each update uses only the new environment and inference does not require its identity.
  • Replay Speech Detector: M-ALRAD jointly processes multi-channel STFTs, learns beamforming weights, and classifies the resulting spectrogram with a CRNN.The detector combines magnitude, sine, and cosine phase features before recurrent processing.
  • Spatial Continual Learning Approach: TSB assigns each environment an independent beamformer head while all heads share one CRNN classifier.During adaptation, the current head and shared classifier are optimized while earlier heads remain frozen.
  • Spatial Continual Learning Approach: At inference, TSB averages logits from all trained heads before softmax, eliminating the need for an environment label.The number of participating heads grows with the environments observed so far.
  • Continual Learning Baselines: The study excludes replay-based methods because retaining speech conflicts with data-minimization, storage-limitation, and erasure requirements, and adapts EWC and GPM as baselines.EWC penalizes movement from prior solutions, whereas GPM projects updates away from retained gradient subspaces.

3. CONTINUAL LEARNING BENCHMARK DESIGN

The benchmark evaluates continual replay detection with EER and sequence-level metrics that capture final error, incremental performance, forgetting, and transfer. These metrics compare sequential learning against single-task and jointly trained references across ordered environments.

  • 3. CONTINUAL LEARNING BENCHMARK DESIGN: EER measures detection performance at each step, while accuracy, forgetting, and transfer metrics summarise the complete incremental sequence.The benchmark evaluates models throughout sequential training rather than only at deployment.
  • 3. CONTINUAL LEARNING BENCHMARK DESIGN: AE reports the mean EER across all environments after the full training sequence, capturing deployment-time error.
  • 3. CONTINUAL LEARNING BENCHMARK DESIGN: AIE averages running error across intermediate steps, rewarding models that perform well throughout training rather than only at the end.
  • 3. CONTINUAL LEARNING BENCHMARK DESIGN: FM measures how much previously learned environments exceed their best earlier EER at the final step, identifying forgetting.A positive FM indicates that the model forgot performance previously achieved on an environment.
  • 3. CONTINUAL LEARNING BENCHMARK DESIGN: BWT compares final EER with EER when each environment was first learned, while IM compares sequential learning with a jointly trained reference.Negative BWT indicates forgetting, and positive IM indicates failure to reach joint-training performance.
  • 3. CONTINUAL LEARNING BENCHMARK DESIGN: FWT compares continual performance on each newly introduced environment with a single-task model trained only on that environment.Positive FWT means earlier environments provided useful forward transfer.

4. EXPERIMENTAL RESULTS

The benchmark evaluates continual replay detection across ReMASC environments, showing that task-specific spatial specialization outperforms standard continual-learning strategies while sequence order strongly affects performance.

  • 4.1. Dataset: The experiments benchmarked four strategies over 24 environment orderings and five runs using ReMASC recordings from four acoustic environments.The study fixed the D2 four-microphone array to isolate acoustic-environment changes.
  • 4.2. Results: 18.79 points of forgetting raised baseline EER on previously learned environments, while TSB significantly reduced AE by 1.02 ± 0.92 points and AIE by 0.81 ± 0.42.TSB was the only method that significantly improved the error-oriented metrics; FM and BWT remained statistically unchanged.
  • 4.2. Results: EWC nearly halved forgetting to 10.18 points versus 18.79 for fine-tuning, but produced +6.18 points of IM and −8.54 points of FWT.Its Fisher-based penalty improved stability while limiting adaptation to new environments.
  • 4.3. Analysis on the environment order: Ending on the moving-vehicle environment produced 28%–33% mean AE, compared with 19%–21% when the indoor lounge environment was last.The vehicle condition is intrinsically more acoustically challenging, and adapting to it last causes over-specialization at the expense of earlier environments.
  • 4.3. Analysis on the environment order: EWC achieved approximately 28% mean AE when the vehicle environment was last, versus 32%–33% for the other methods.This partial robustness reflects reduced weight drift, but it comes at the cost of plasticity.

5. CONCLUSION

The paper introduces a continual-learning benchmark for multi-channel replay detection across acoustic environments and compares four exemplar-free strategies over all orderings. TSB improves final and incremental accuracy through spatial specialization, while the final environment has a stronger effect on final EER than the algorithm choice.

  • 5. CONCLUSION: TSB improves final and incremental accuracy without reducing forgetting, indicating that its gains come from spatial specialization rather than added stability.The benchmark evaluates four exemplar-free strategies over 24 orderings and five seeds.
  • 5. CONCLUSION: The environment closing the sequence drives final EER far more than the algorithm does.None of the evaluated strategies removes catastrophic forgetting.
Loading 2609.11194v1…