PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Shared Encoder in Spoof-Aware Speaker Verification with Hybrid Wavelet Prompt Tuning

Aref Farhadipour, Srikanth Madikeri, Teodora Vukovic, Volker Dellwo, Petr Motlicek

arXiv:2610.08845v1eess.AScs.SD

TL;DR

SASV systems must verify speaker identity and speech genuineness, but cascades use separate large encoders and single-embedding end-to-end models face conflicting objectives. The paper uses task-specific hybrid prompts and heads with a shared frozen W2V-BERT 2.0 backbone. It reports results on SpoofCeleb and ASVspoof5, including ASVspoof5 evaluation without external training data.

  • Problem

    SASV must assess speaker identity and liveness, while cascades use separate backbones and end-to-end systems can struggle to optimize one representation for both objectives.

    Source: §I · pp. 1–2

  • Method

    USE adapts a frozen W2V-BERT 2.0 encoder with task-specific hybrid wavelet prompts and separate ASV and CM heads whose scores are combined at inference.

    Source: §I · pp. 1–2, §II · p. 2

  • Results

    SpoofCeleb evaluation reached 0.03% CM-EER, while ASVspoof5 evaluation reached 0.092 min a-DCF without external training data.

    Source: §IV · p. 6

  • Takeaways & Limitations

    The frozen shared encoder permits independent addition of task-specific heads without retraining or modifying existing modules.

    Source: §I · p. 2, C. Ablation and Scalability · p. 5

Abstract

from arXiv · show

Spoofing-aware speaker verification (SASV) must confirm who is speaking and that the speech is genuine, but current systems gain one at the expense of the other. Modular cascades are accurate yet run two large self-supervised encoders, whereas end-to-end models are efficient but lose accuracy when a single embedding has to serve two conflicting objectives. We show that this trade-off between efficiency and specialization is avoidable. A single frozen W2V-BERT~2.0 backbone is adapted per task by deep hybrid Wavelet Prompt Tuning (WPT), so each branch obtains its own view of the shared encoder through dedicated prompts and a task head, and the scores are combined only at inference. Training only about 7M parameters, a small fraction of those a strong two-encoder cascade requires, the system surpasses it on SpoofCeleb evaluation with 0.03% CM-EER and 0.038 min a-DCF against 0.16% and 0.047. Because the backbone is shared and frozen, new branches attach without retraining the heads and prompts already in place. On ASVspoof5, with adversarial attacks and codec distortions, it reaches 0.092 min a-DCF and 3.60% CM-EER using only the provided data, indicating that the design holds under realistic in-the-wild threats.

I. INTRODUCTION

SASV must verify speaker identity and speech genuineness, but existing approaches face an efficiency–specialization trade-off. USE addresses it with a frozen shared encoder and task-specific prompt-conditioned branches.

I. INTRODUCTION

  • SASV integrates ASV and countermeasure systems to address spoofing threats including high-fidelity TTS, voice conversion, and neural deepfakes.The motivation is the vulnerability of speaker verification systems to spoofing despite their precision in distinguishing target from non-target speakers.Source: §I · p. 1
  • Two-backbone cascades incur computational redundancy and are difficult to extend, while single-embedding end-to-end models often underperform on SASV.Cascades also risk error propagation because independently optimized identity and liveness representations do not interact.Source: §I · p. 1
  • USE pairs a frozen W2V-BERT 2.0 backbone with separate ASV and CM heads conditioned by parameter-efficient hybrid Wavelet Prompt Tuning.The approach extends wavelet prompt tuning from frozen XLSR deepfake detection to a shared encoder serving both tasks.Source: §I · p. 1
  • Task-specific prompts give ASV and CM distinct conditioned contexts over shared encoder knowledge, avoiding a single embedding for conflicting objectives.The branches' scores are combined at inference, and new task heads can be added without retraining existing modules.Source: §I · p. 2
  • Spoofing attacks increasingly mimic speaker spectral and prosodic traits, motivating aggregation across transformer layers to capture both acoustic artifacts and identity cues.The approach is validated on SpoofCeleb and ASVspoof5, which include environmental noise, reverberation, codec, and channel distortions.Source: §I · p. 2

II. PROPOSED METHOD

USE shares a frozen W2V-BERT 2.0 encoder across speaker verification and spoof detection while adapting each task through dedicated prompts and heads. Hybrid Wavelet Prompt Tuning transforms wavelet prompts and injects both prompt banks at every layer, where audio states attend to the prompt-augmented sequence.

II. PROPOSED METHOD

  • USE combines speaker verification and spoof detection through a shared W2V-BERT 2.0 encoder with multi-layer features.The backbone has 24 Conformer layers and approximately 600M parameters, pretrained on 4.5 million hours of diverse speech.Source: §II · p. 2

A. Hybrid Wavelet Prompt Tuning

  • Regular and wavelet prompts are injected together at every transformer layer, with audio frames querying the prompt-augmented sequence.Prompt positions are discarded after each block update, and fresh layer-specific prompts are injected at the next layer.Source: Figure 1 · p. 2, A. Hybrid Wavelet Prompt Tuning · p. 3
  • Each layer receives task-specific regular and wavelet prompt banks, while all W2V-BERT 2.0 backbone weights remain frozen.The banks contain 15 regular and 8 wavelet tokens per layer; they are the prompt module's only trainable quantities.Source: A. Hybrid Wavelet Prompt Tuning · p. 2
  • The wavelet bank is transformed with a Haar DWT over its token–channel matrix, then its sub-bands are concatenated and reshaped into the prompt size.The transform acts on prompt tokens, not audio time or frequency.Source: A. Hybrid Wavelet Prompt Tuning · p. 3
  • Each independently trained task branch combines its prompts with a head that reads all 24 layers and produces a 256-dimensional embedding.Both heads use attentive temporal pooling and a Refiner bottleneck module.Source: A. Hybrid Wavelet Prompt Tuning · p. 3

B. ASV Head: Layer-Fused MHFA

The ASV head combines layer-wise backbone features into a speaker embedding using separate learned weights for key and value streams. It then pools and refines the representation for speaker verification.

B. ASV Head: Layer-Fused MHFA

  • Separate learnable weights combine the 24 transformer layers into fused key and value streams for the ASV head.The fixed weighted-sum strategy is intended to reflect speaker identity as a globally stable attribute and yield a more stable, lower-variance embedding than input-dependent fusion.Source: B. ASV Head: Layer-Fused MHFA · p. 3
  • Multi-head attentive temporal pooling, a Refiner module, and an embedding MLP map the fused streams to a 256-dimensional speaker embedding.The pooling uses keys derived from the fused key stream and queries and values from the fused value stream.Source: B. ASV Head: Layer-Fused MHFA · p. 3
  • The head is trained with ArcFace loss, and inference scores enrollment–test pairs by cosine similarity between their L2-normalized embeddings.The cited passage specifies ArcFace margin m=0.3 and scale s=30.Source: B. ASV Head: Layer-Fused MHFA · p. 3

C. CM Head: Cross-Layer Attention with DSU

The CM head fuses layer outputs into a frame sequence and applies feature-statistics perturbation during training before pooling and bona fide/spoof classification.

C. CM Head: Cross-Layer Attention with DSU

  • The CM head combines layer outputs at each frame into a fused sequence, with adaptive recalibration suited to spoofing cues that occur at different abstraction levels.The passage contrasts this adaptive fusion with the ASV head’s static layer weighting.Source: C. CM Head: Cross-Layer Attention with… · p. 3
  • During training, Domain Shifts with Uncertainty perturbs the fused feature statistics with probability 0.3 before temporal pooling to improve robustness to unseen attacks and codec mismatch.The pooled sequence passes through an 8-head attentive temporal pool, Refiner bottleneck, and embedding MLP for binary bona fide/spoof classification.Source: C. CM Head: Cross-Layer Attention with… · p. 3
  • Weighted cross-entropy uses class weights [10, 1] for bona fide and spoof classes, respectively, to counter the spoof-heavy training imbalance.The passage relates the larger bona fide weight to the 10 attack types in the SpoofCeleb training set.Source: C. CM Head: Cross-Layer Attention with… · p. 4

D. SASV Score Integration

The integration rule calibrates ASV and CM outputs to a common range and multiplies them to produce the final SASV score.

D. SASV Score Integration

  • ASV cosine similarities range from −1 to 1, whereas CM produces bounded posterior scores, so their outputs require calibration before integration.The ASV score is sigmoid-calibrated to [0, 1], while the CM head supplies the bona fide posterior probability in [0, 1].Source: D. SASV Score Integration · p. 4
  • The final SASV score is the product of the sigmoid-calibrated ASV score and the CM bona fide posterior.The product acts as an AND-gate, becoming high only when both tasks accept the trial.Source: D. SASV Score Integration · p. 4
  • At inference, task-specific prompt-and-head branches make two forward passes through the shared 600M-parameter backbone, and their scores are combined only then.The product rule is fixed, parameter-free, and trial-independent, unlike cohort normalization.Source: D. SASV Score Integration · p. 4

E. The SASV Datasets

The study evaluates SASV on SpoofCeleb and ASVspoof5, benchmarks with different speaker, utterance, and spoofing conditions.

E. The SASV Datasets

  • SpoofCeleb is the primary benchmark, derived from VoxCeleb1 and preserving in-the-wild noise, reverberation, and channel distortions.It contains 1,251 speakers, about 150k bona fide utterances, and roughly 2.4 million spoofed utterances generated by 23 contemporary synthesis systems.Source: E. The SASV Datasets · p. 4
  • ASVspoof5 Track 2 includes 1,922 speakers and about 680,000 utterances, with 32 sophisticated spoofing attacks and seven codec conditions in its evaluation set.The cited passage names Opus and Speex among the codec conditions.Source: E. The SASV Datasets · p. 4
  • Results are reported using EER and min a-DCF under the official Wildspoof 2026 and ASVspoof 2024 challenge settings.The specified cost parameters are Cmiss=1 and Cfa=10, with priors πtar=0.9405, πnon=0.0095, and πspf=0.05.Source: E. The SASV Datasets · p. 4

III. EXPERIMENTAL RESULTS

The evaluation establishes SpoofCeleb performance benchmarks before identifying the optimal configuration for subsequent ASVspoof5 evaluation.

III. EXPERIMENTAL RESULTS

  • The experiments first establish SpoofCeleb benchmarks, then train and evaluate the optimal configuration on ASVspoof5.The section introduces two baseline systems before detailing the proposed architecture.Source: §III · p. 4

A. Baseline Systems

The section compares an end-to-end SASV baseline with a modular cascade of separate countermeasure and speaker-verification systems. The cascade performs better than the end-to-end baseline but requires two SSL models and is difficult to extend.

A. Baseline Systems

  • The two baselines represent an end-to-end shared-backbone strategy and a cascade in which countermeasure screening precedes speaker verification or rejection.Source: A. Baseline Systems · p. 4
  • The end-to-end baseline uses SKA-TDNN with jointly trained speaker-identification and bona fide/spoof branches to learn shared spoof-aware speaker embeddings.It is trained on 400-frame SpoofCeleb chunks for 50 epochs.Source: A. Baseline Systems · p. 4
  • The end-to-end baseline records 0.256 min a-DCF on development and 0.230 on evaluation.Source: A. Baseline Systems · p. 5
  • The cascade pairs a WPT-XLSR/AASIST countermeasure using last-layer features with a pretrained WavLM-ECAPA-TDNN speaker-verification component.The speaker-verification component is optimized on VoxCeleb2.Source: A. Baseline Systems · p. 5
  • 0.047 min a-DCF on evaluation (0.043 on development) lets the modular cascade surpass the end-to-end baseline, which records 0.230 and 0.256, respectively.The cascade totals 320 million trainable parameters and lacks modular scalability without adding another backbone.Source: A. Baseline Systems · p. 5

B. USE Configuration

USE combines task-specific heads and prompts with a frozen shared backbone, yielding strong SpoofCeleb results and a parameter-efficient modular design. Ablations and an added ASV branch examine component contributions and scalability.

B. USE Configuration

  • The independently trained ASV and CM heads contain 1.5M and 5.5M trainable parameters, respectively, and use 23 ASV prompts across 24 layers.Both heads were trained for 10 epochs without joint multi-task backpropagation.Source: B. USE Configuration · p. 5
  • The ASV head uses the full backbone depth, weighting layers 17–21 heavily, while CM concentrates on layers 2, 3, 19, 20, and 21.The reported interpretation is that ASV benefits from deeper semantic features, whereas CM combines early spectral patterns with later dependencies.Source: B. USE Configuration · p. 5
  • 0.038 min a-DCF and 3.24% SASV-EER are the final SpoofCeleb results, with CM EER of 0.03% and ASV EER of 3.65%.The system tunes around 1% of total parameters and ranks second on the official SpoofCeleb leaderboard.Source: B. USE Configuration · p. 5

C. Ablation and Scalability

  • Replacing CM cross-layer attention with a weighted sum reduces parameters from 7.0M to 3.2M while eval min a-DCF shifts from 0.038 to 0.039.Removing the Refiner yields 0.055 eval min a-DCF; omitting multi-layer integration yields 4.93% Eval SASV-EER. Removing prompts gives 0.086 eval min a-DCF, regular prompts 0.044, and USE 0.038, with CM-EER improving from 0.15% to 0.03% at 7.0M parameters.Source: C. Ablation and Scalability · p. 5
  • A VoxCeleb2-trained ASV head with 23 exclusive prompts adds 1.5M learnable parameters without altering existing modules.The expanded system combines the ASV heads by score averaging and then combines them with CM by dot product; it reaches 2.94% and 2.73% SASV-EER on development and evaluation, respectively, and 0.035 min a-DCF.Source: C. Ablation and Scalability · pp. 5–6

D. Evaluation on ASVspoof5 Dataset

For ASVspoof5, the framework uses codec augmentation and achieves 0.092 min a-DCF on evaluation. It ranks second in the open condition and above all systems in the closed condition without extra training data.

D. Evaluation on ASVspoof5 Dataset

  • All seven evaluation codecs are used for training augmentation, each with per-utterance probability p = 0.35.The benchmark includes diverse adversarial attacks and transmission codecs.Source: D. Evaluation on ASVspoof5 Dataset · p. 6
  • 0.092 min a-DCF on evaluation places USE second on the open-condition leaderboard and above all systems in the closed condition.The system uses only the provided corpus and a W2V-BERT 2.0 backbone.Source: D. Evaluation on ASVspoof5 Dataset · p. 6

E. ASVspoof5 Codec and Attack Breakdown

On ASVspoof5, the system achieves 0.092 min a-DCF and 3.60% aggregate CM-EER; codec errors vary less than attack errors. The conclusion reports parameter-efficient performance across benchmarks.

E. ASVspoof5 Codec and Attack Breakdown

  • 2.88%–3.49% CM-EER across codecs C01–C10 rises to 4.48% on the most compressed condition, C11.Codec variability accounts for part of evaluation CM error, but its spread is relatively narrow compared with attack-wise variation.Source: E. ASVspoof5 Codec and Attack Breakdown · p. 6
  • 0.48% and 0.76% CM-EER on A29 and A17 contrast with 7.74% on A28 and 4.97% on A31.The attack-wise gap helps explain why aggregate ASVspoof5 CM-EER exceeds SpoofCeleb’s result.Source: E. ASVspoof5 Codec and Attack Breakdown · p. 6

IV. CONCLUSION

  • Tuning less than 1% of total parameters, the framework surpasses resource-intensive cascaded and end-to-end architectures.The conclusion attributes its capture of distinct ASV and CM layer-wise distributions to multi-layer aggregation.Source: §IV · p. 6
  • 0.092 min a-DCF on ASVspoof5 accompanies 3.60% aggregate CM-EER pooled across seven codecs and sixteen attack types.The conclusion also reports 0.03% CM-EER on SpoofCeleb and 0.035 min a-DCF in the expanded modular configuration.Source: §IV · p. 6, E. ASVspoof5 Codec and Attack Breakdown · p. 6
Loading 2610.08845v1…