PaperlayerView exact revision on arXiv

Source-linked AI summary

Spooftral: Can Voxtral Audio-Language Model Detect Speech Spoofing?

Avishai Weizman, Yehuda Ben-Shimol, Itshak Lapidot

arXiv:2609.28713v1eess.AScs.CL

TL;DR

Spoofing countermeasures still face generalization challenges across unseen attacks and changing speech conditions. This work evaluates Voxtral with instruction-guided label-likelihood scoring and task-specific adaptation, finding that frozen LLM processing reduces cue separability while adapted Spooftral variants outperform the compared approaches.

  • Problem

    Spoofing countermeasures continue to face generalization challenges on unseen attacks and distribution shifts in benchmark speech.

    Source: §I · p. 1

  • Method

    The study frames spoofing detection as an instruction-guided task that scores bonafide and spoof label sequences by length-normalized likelihood, and adapts Voxtral with DoRA.

    Source: §I · p. 1, §III · p. 2

  • Results

    Spooftral variants outperform Spooftral-Enc on most evaluation sets and both outperform the reported Whisper-based approach, while frozen LLM representations are less separable than audio-adapter representations.

    Source: §V · pp. 4–5

  • Takeaways & Limitations

    The results support task-specific adaptation for Voxtral spoof detection, with adapted LLM representations complementing the acoustic encoder.

    Source: §V · pp. 5–6

Abstract

from arXiv · show

Self-supervised learning (SSL) countermeasures (CMs) have shown strong performance in recent years. However, they often show degraded performance while facing unseen spoofing attacks and mismatched conditions. This study examines the Voxtral audio-language model (ALM) framework for spoofing detection, as a step toward combining CM capabilities within the ALM framework. We analyze how Voxtral captures spoofing cues through audio-text processing and propose an instruction-guided approach that uses label-sequence likelihoods to evaluate bonafide and spoofed speech. Experiments on the ASVspoof databases show that without task-specific adaptation, the LLM layers emphasize semantic representations, reducing the separability of spoof-discriminative acoustic cues compared to the Whisper-based audio encoder. Consequently, spoofing-related information becomes less separable after language-model processing. We also applied lightweight adaptation using weight-decomposed low-rank adaptation (DoRA) to the Voxtral model and propose the Spooftral model, achieving an equal error rate (EER) of 4.25% on the ASVspoof5 evaluation set.

I. INTRODUCTION

ASV systems face replay, TTS, and voice-conversion spoofing, while countermeasures still struggle to generalize to unseen attacks and shifting conditions. This work studies spoofing detection within Voxtral, analyzing cue propagation and adapting the model for the task.

I. INTRODUCTION

  • ASV systems used for biometric applications remain vulnerable to replay, TTS, and voice-conversion attacks, motivating countermeasures for spoofing detection –.Source: §I · p. 1
  • Unseen spoofing attacks remain a generalization challenge, and ASVspoof5 adds harder attacks and a shift in bonafide-speech distributions relative to ASVspoof2019, [10]–.These changes motivate spoofing detectors that work across attacks and conditions.Source: §I · p. 1
  • The study analyzes how spoof-discriminative cues propagate through Voxtral and compares embeddings before and after its frozen LLM layers.It also evaluates the Whisper-based audio encoder on ASVspoof databases.Source: §I · pp. 1–2
  • The authors formulate spoofing detection as an instruction-guided ALM task and adapt Voxtral using lightweight DoRA training [32], [34]–.Source: §I · p. 1
  • The work takes a step toward a unified ALM framework for speech understanding and spoofing detection without an external countermeasure [31]–[34].Source: §I · p. 1

II. RELATED WORK

Prior work spans multiple audio-language architectures and reports both promise and limitations for spoofing detection. This paper focuses on Voxtral and the limited evidence about how spoof cues change across its processing stages.

II. RELATED WORK

  • ALMs include Two-Tower, Two-Head, One-Head, and cooperative designs, which differ in how they encode and fuse audio and language [31]–.Source: §II · p. 2
  • Voxtral is a Two-Head ALM with a Whisper Large-v3 audio encoder, audio-adapter downsampling, and a pretrained LLM, [35].Its training includes pre-training, supervised fine-tuning, and preference alignment such as DPO, [47].Source: §II · p. 2
  • Prior findings report quantization sensitivity in ALM countermeasures and strong Qwen-Audio spoof-detection performance on ASVspoof2019 and In-the-Wild, [10], [34],.Source: §II · p. 2
  • The Spooftral overview depicts joint audio-text processing followed by likelihood-based spoofing decisions.Source: Figure 1 · p. 2
  • Prior ALM-based countermeasure studies emphasize end-to-end detection, with limited investigation of how spoof-discriminative information propagates across model stages.Source: §II · p. 2

III. GENERATIVE LABEL-LIKELIHOOD CLASSIFICATION FOR SPOOFING DETECTION

The section casts spoofing detection as instruction-guided classification using label-sequence likelihoods, with a fixed prompt and fixed class labels for reproducible decisions. It derives an inference score from the normalized likelihood difference between the bonafide and spoof hypotheses and trains with class-weighted cross-entropy, optionally adding a margin term.

III. GENERATIVE LABEL-LIKELIHOOD CLASSIFICATION FOR SPOOFING DETECTION

  • A fixed prompt and fixed label tokens make the generative classification decision controlled and reproducible.The paper uses meaningful class labels rather than arbitrary symbols, which may let the model associate acoustic cues with pretrained semantic representations.Source: §III · p. 2
  • At inference, Voxtral compares length-normalized likelihoods for “bonafide” and “spoof” label sequences, using their score difference as the detection score.The framework uses raw logits without sampling or temperature scaling.Source: §III · pp. 2–3
  • Label-sequence log-likelihoods are normalized by token length to avoid potential length bias, and the framework permits other fixed prompt-label interfaces.Any alternative interface must be used consistently during training and inference.Source: §III · p. 3
  • Training uses class-weighted cross-entropy over the two label-sequence scores and can add margin-based regularization to promote separation between competing hypotheses.The final loss combines cross-entropy and the margin term; the margin handles cases where the class-oriented score difference falls below a predefined margin.Source: §III · p. 3

IV. DATABASES AND EVALUATION

The section introduces the databases and evaluation metric used in the study. Table I reports bonafide and spoofed utterance counts for four ASVspoof databases.

IV. DATABASES AND EVALUATION

  • The section describes the databases and evaluation metric used in the study.Source: §IV · p. 3
  • Table I covers ASVspoof2019 LA, ASVspoof2021 LA, ASVspoof2021 DF, and ASVspoof5.Its subject is the number of utterances in each database.Source: Table I · p. 3
  • Table I distinguishes bonafide and spoofed utterance counts for the listed databases.Source: Table I · p. 3

A. ASVspoof2019 LA Database

ASVspoof2019 LA includes VC and TTS attacks across its training, development, and evaluation sets, with clean recording conditions. Its evaluation set adds unseen and overlapping attacks.

A. ASVspoof2019 LA Database

B. ASVspoof 2021 LA and DF Databases

ASVspoof2021 provides evaluation-only LA and DF partitions, so participants rely on ASVspoof2019 LA training and development data. The two partitions apply different codec conditions.

B. ASVspoof 2021 LA and DF Databases

C. ASVspoof5 Database

ASVspoof5 uses distinct attack sets for training and development, then evaluates on unseen attacks under additional transmission and codec conditions. Performance is assessed with EER.

C. ASVspoof5 Database

  • The training set contains eight TTS attacks, while the development set contains eight different attacks.Source: C. ASVspoof5 Database · p. 3
  • The evaluation set contains 16 unseen attacks, including adversarial attacks, and applies transmission and compression codecs to bonafide and spoofed speech.Source: C. ASVspoof5 Database · p. 3
  • EER is computed from log-likelihood differences for Spooftral and classifier scores for linear probing and Spooftral-Enc.Source: C. ASVspoof5 Database · p. 3

V. EXPERIMENTS AND RESULTS

Experiments compare spoof cues across Voxtral processing stages and test instruction-guided, DoRA-based adaptation. Spooftral performs competitively across ASVspoof databases, while frozen LLM representations show weaker spoof separability than audio-adapter representations.

V. EXPERIMENTS AND RESULTS

  • Linear probing mean-pools frozen audio-adapter or LLM hidden states and trains only a linear binary classifier to assess encoded spoof-discriminative information.Text-generation EER was considered unreliable because an unadapted model may use different textual responses for the same spoof decision.Source: §V · p. 4
  • On a fixed prompt, linear probes perform worse after the LLM decoder than after the audio adapter, consistent with semantic pretraining rather than spoof-detection training.The authors attribute this difference to Voxtral's pretraining for semantic tasks such as ASR and speech-to-text translation.Source: §V · p. 4, Table II · p. 4
  • PaCMAP projections show clearer spoof separation after the audio adapter than after the LLM layers, with the adapter advantage also present on development and evaluation sets.Source: §V · p. 4, Figure 2 · p. 4
  • Spooftral adapts Voxtral with DoRA, while Spooftral-Enc omits the LLM decoder; both keep the audio adapter trainable and use audio perturbations during training.For ASVspoof5, the attention-DoRA configuration uses rank 32 and α = 64.Source: §V · pp. 4–5
  • ASVspoof5's different training and development attacks may encourage generalizable representations, and its development and evaluation EERs suggest good generalization across attack types.The authors contrast this design with ASVspoof2021, whose training and development sets share attack types while evaluation uses unseen attacks.Source: §V · p. 5
  • Spooftral outperforms Spooftral-Enc on most evaluation sets, and both variants outperform the Whisper-based approach using lightweight adaptation without fusion or extra spoofing data.The Table III caption identifies the cross-database comparison; the paper also reports competitive ASVspoof5 performance against other SSL-based countermeasures.Source: §V · pp. 5–6, Table III · p. 5, Table V · p. 5
  • The bonafide/spoof interface yields lower EERs than real/fake, especially on development, and margin regularization lowers EER for Attention DoRA on both ASVspoof5 sets.The authors note that further study is needed to separate prompt effects from target-label-sequence effects.Source: §V · pp. 5–6, Table IV · p. 5
  • Per-attack detection is weakest for A13 and A15 on development and for A28 on evaluation, using each set's EER threshold.The authors note that other analyses also identified A28 as challenging for countermeasures.Source: §V · p. 6, Figure 3 · p. 6

VI. DISCUSSION AND CONCLUSIONS

Frozen Voxtral representations lose spoof-cue separability after LLM decoding, while task-adapted Spooftral can use the decoder for detection. The paper reports competitive ASVspoof5 performance and outlines directions for joint speech tasks and improved audio representations.

VI. DISCUSSION AND CONCLUSIONS

  • 4.25% EER on the ASVspoof5 evaluation set is competitive with ASVspoof5 Challenge submissions.The authors suggest the larger, more diverse ASVspoof5 training set may support more robust spoof-discriminative representations and better generalization.Source: §VI · p. 6
  • In the frozen setting, spoof cues are more separable before the LLM decoder, whose semantic-robustness objective reduces fine-grained acoustic-artifact separability.After DoRA adaptation, the decoder can contribute effectively to spoof detection.Source: §VI · p. 6
  • Future work targets joint spoofing and speech-task handling, more artifact-sensitive encoders, alternative adaptation and training, and reasoning strategies for generalization.The paper notes that current ALM frameworks use a Whisper-based audio encoder.Source: §VI · p. 6
Loading 2609.28713v1…