PaperlayerView exact revision on arXiv

Source-linked AI summary

Textual Acoustic Grounding for Generalizable LLM-Based Deepfake Voice Detection

Yassine El Kheir, Xin Wang, Wanqing Ge, Tim Polzehl, Sebastian Moeller, Junichi Yamagishi

arXiv:2608.30622v1cs.SDeess.AS

TL;DR

Deepfake voice detection generalizes poorly to unseen types, while ALLMs struggle to connect detailed acoustic representations with LLM semantic space. The paper benchmarks audio encoders with Qwen LLMs and introduces structured textual grounding from openSMILE acoustic features. This improves out-of-domain robustness, supports a strong frozen-LLM baseline, and makes LoRA adaptation more effective.

  • Problem

    Deepfake voice detectors struggle to generalize from fixed training data to unseen deepfake types, while mapping audio representations into LLM semantic space remains underexplored.

  • Method

    The paper benchmarks diverse audio encoders with Qwen LLMs and injects openSMILE acoustic features as structured text tokens for cross-modal grounding.

  • Results

    The framework achieves robust out-of-domain detection, with audio-encoder fine-tuning driving generalization and textual grounding mitigating LLM fine-tuning degradation.

  • Takeaways & Limitations

    A frozen LLM paired with a fine-tuned audio encoder provides an efficient robust baseline, while openSMILE grounding enables more effective LoRA adaptation.

Abstract

from arXiv · show

Deepfake voice detection suffers from poor generalization across unseen domains. While Audio Large Language Models (ALLMs) show promise, the modality gap between continuous audio embeddings which capture the subtle acoustic details necessary for deepfake detection and the semantic space of LLMs remains a critical, underexplored bottleneck. We address this by benchmarking diverse audio encoders integrated with Qwen LLMs (0.5B to 7B parameters). First, we demonstrate that fine-tuning the LLM alone risks out-of-domain overfitting, making a frozen LLM a stronger, resource-efficient baseline. Second, to explicitly bridge the modality gap, we introduce a cross-modal prompting strategy that injects linguistic-knowledge-driven acoustic features (via openSMILE) as structured text tokens. This explicit textual grounding not only enhances the frozen baseline but also makes LLM fine-tuning more effective. Ultimately, our approach demonstrates state-of-the-art resilience on the out-of-domain ITW and MLAAD benchmarks, yielding over \textbf{16.2\%} absolute improvement in Macro-F1 over existing ALLM baselines while maintaining competitive in-domain performance. All models reported in this work are \href{https://huggingface.co/01Yassine/AudioLLM-Deepfake-Detection}{publicly available}.

I. INTRODUCTION

The paper targets robust deepfake voice detection across unseen types and studies how audio-encoder integration, LLM adaptation, and textual acoustic grounding affect generalization.

  • Robust detection remains difficult when models trained on fixed data encounter unseen deepfake types.
  • ALLMs offer potential for acoustic anomaly detection, but mapping audio representations into LLM semantic space remains a critical alignment challenge.
  • The study compares audio encoders and LLM integration strategies while asking whether out-of-domain robustness can be achieved without fine-tuning the LLM backbone.
  • It investigates whether explicit textual descriptions of acoustic cues enhance a frozen baseline and interact with LLM adaptation strategies.
  • The proposed features come from openSMILE and cover acoustic properties that are fed to the LLM in structured text format.
  • Fine-tuning the audio encoder drives generalization, whereas fine-tuning the LLM alone degrades out-of-domain performance; textual acoustic prompting mitigates this degradation and makes LoRA adaptation synergistic.

II. RELATED WORK

Prior work established ALLM-based deepfake detection but left audio-encoder design and its interaction with LLM training underexplored; this paper adds systematic encoder comparison and textual acoustic prompting.

  • Earlier ALLM systems used pretrained models or examined limited encoder choices, leaving broader audio-encoder design insufficiently explored.
  • The study systematically compares Whisper, Wav2Vec2, EAT, and speech codec encoders, including codecs not previously investigated for deepfake voice detection.
  • The interaction between audio-encoder choice and LLM-backbone training strategy remains an unexplored research question addressed by this work.
  • The paper introduces openSMILE-derived text descriptions of acoustic features covering cues beyond time-frequency representations, including intonation.
  • Using descriptive acoustic prompts to help LLMs detect synthetic speech artifacts is presented as a novel approach.

III. PROPOSED SYSTEM

The proposed ALLM detector projects encoded audio into the LLM input space, combines it with a text prompt, and generates a real/fake answer.

  • Architecture: The framework consists of an audio encoder Fenc, a lightweight linear projector P, and an LLM backbone F.
  • Architecture: For utterance x, the system computes Taud = P(Fenc(x)) ∈ R^T×D, where T is the token count and D is the vector dimension.
  • Architecture: The projector is fine-tuned alongside the audio encoder even when the LLM backbone remains frozen.
  • Architecture: Audio embeddings are concatenated with text-prompt embeddings, and the LLM produces a real/fake output surrounded by <answer> tags.
  • The experiments study both audio-encoder integration and whether knowledge-driven acoustic features improve detection.

B. Audio Encoder and LLM Configuration

The paper evaluates speech, environmental-audio, and neural-codec encoders, comparing frozen, weighted-aggregation, and end-to-end fine-tuned integration regimes.

  • Choice of Audio Encoder and Fine-Tuning Strategy: Whisper may discard non-linguistic acoustic details because it was trained for speech recognition, motivating alternatives for deepfake detection.
  • Choice of Audio Encoder and Fine-Tuning Strategy: Neural codecs encode speech into quantized vectors, but whether these features remain useful for deepfake voice detection is an open question.
  • Choice of Audio Encoder and Fine-Tuning Strategy: The evaluated encoder set includes Whisper, Wav2Vec2, EAT, and neural audio codecs such as DAC and SpeechTokenizer.
  • Choice of Audio Encoder and Fine-Tuning Strategy: Wav2Vec2 preserves acoustic features across linguistic levels through contrastive self-supervised learning over raw speech waveforms.
  • Choice of Audio Encoder and Fine-Tuning Strategy: EAT represents a non-speech self-supervised encoder pretrained on large-scale environmental sound data.
  • Choice of Audio Encoder and Fine-Tuning Strategy: The study compares freezing, weighted aggregation across encoder blocks, and fine-tuning all encoder parameters using gradients from the LLM backbone.

2) Choice of LLM Backbone and Fine-Tuning Strategy:

The paper uses Qwen2.5-Instruct models as binary real/fake detectors and explores frozen or LoRA-adapted backbones. It bridges audio and semantic spaces by serializing 88 openSMILE acoustic features into descriptive text prompts.

  • LLM Backbone: The study also evaluates Qwen2.5-Omni-3B with full LoRA adaptation as a native end-to-end multimodal baseline.
  • Textual Acoustic Grounding: Because LLMs may overlook acoustic details and neural representations lack explicit interpretability, the method adds expert-validated openSMILE eGeMAPSv2 features.The selected feature set contains 88 dimensions.
  • Textual Acoustic Grounding: Continuous openSMILE vectors are serialized into descriptive text, expanding acronyms such as F0 and F1 for interpretability.The resulting prompt adds approximately 1500 tokens.
  • Textual Acoustic Grounding: Textual acoustic features let the LLM compare and evaluate forensic evidence without changing the LLM backbone architecture.

IV. EXPERIMENT SETUP

The experiments compare five audio encoder families on ASVspoof and out-of-domain benchmarks using Qwen-based binary classification. Performance is reported with Accuracy and Macro F1, while Table I separates benchmark-level F1 from overall averages.

  • Audio Encoders: The pipeline evaluates Wav2Vec2.0 XLS-R, Whisper, EAT, SpeechTokenizer, and DAC as audio encoder families.For codecs, continuous embeddings come from the first four or six residual vector-quantization codebooks.
  • Metrics: Accuracy and Macro F1 are reported to match the generative classification setting in which the LLM outputs real or fake text tokens.

V. EXPERIMENT PART I: AUDIO ENCODER AND LLM CONFIGURATION

The first experiment tests encoder choice and integration strategy for binary deepfake detection, emphasizing out-of-domain robustness without necessarily fine-tuning the LLM. Encoder fine-tuning consistently provides the strongest generalization, especially for Whisper and Wav2Vec2.

  • Experimental Aim: The first experiment evaluates audio encoder integration for binary real/fake detection while excluding openSMILE features.
  • Experimental Design: The study compares Qwen2.5-Instruct scales of 0.5B, 3B, and 7B, five encoders, frozen or LoRA-16 LLMs, and three encoder integration regimes.LoRA-16 uses rank 16 and α = 64.
  • Encoder Fine-Tuning: 75.60% ITW F1 and 74.24% ML F1 are achieved by fine-tuned Wav2Vec2 with LoRA-16, up from 54.95% and 69.78% with a frozen encoder.The frozen encoder still reaches 99.44% LA19 F1, illustrating the in-domain/out-of-domain contrast.
  • Encoder Fine-Tuning: 76.46% ITW F1 is reached by fine-tuned Whisper with a frozen LLM, compared with 65.73% for its frozen-encoder counterpart under LoRA-16.
  • Integration Strategy: 55.00% ITW F1 from weighted Whisper aggregation with a frozen LLM trails the 76.46% achieved by fully fine-tuning Whisper.
  • Encoder Choice: Whisper and Wav2Vec2 outperform EAT and neural codecs out of domain, with fine-tuned EAT reaching only 54.20% ITW F1 and 69.91% ML F1.Under the same FT+Frozen regime, Whisper reaches 76.46% ITW and 80.45% ML, while Wav2Vec2 reaches 74.79% and 78.98%.
  • Encoder Choice: 65.83% and 65.59% ML F1 are obtained by SpeechTokenizer and DAC with frozen encoders and LoRA-16, the weakest codec results.The paper suggests vector quantization may discard inaudible artifacts useful for detection.

C. Results on LLM Backbone and Fine-Tuning

The results show that a frozen LLM paired with a fine-tuned audio encoder is a strong baseline, whereas LLM adaptation and brute-force scaling are not consistently beneficial. The next experiment therefore motivates explicit textual acoustic grounding to support effective LLM adaptation.

  • Frozen LLM Baseline: 91.81% average Accuracy and 91.09% average F1 are achieved by fine-tuned Whisper with a frozen LLM, the highest averages among the reported Whisper configurations.Fine-tuned Wav2Vec2 with a frozen LLM reaches 89.75% Accuracy and 88.43% F1.
  • LLM Fine-Tuning: LoRA adaptation reduces Whisper FT ML F1 from 90.50% to 68.18% compared with a frozen LLM.
  • LLM Fine-Tuning: A frozen LLM consistently outperforms LoRA-tuned counterparts across tested LoRA alpha values and 0.5B, 3B, and 7B backbones.
  • LLM Scaling: 91.09% average F1 at 0.5B declines slightly to 90.94% at 3B and 90.58% at 7B for Whisper FT with a frozen LLM.
  • Summary: Whisper and Wav2Vec2 are preferred over EAT and neural codecs, while a frozen instruct-LLM with a fine-tuned SSL encoder offers a resource-efficient alternative.
  • Motivation: The findings motivate testing whether text-based acoustic features can bridge the modality gap and make LLM adaptation more effective.

VI. EXPERIMENT PART II: EFFECTIVENESS OF OPENSMILE FEATURE INJECTION

This section evaluates whether structured openSMILE text prompts bridge acoustic features and Qwen LLMs across encoder families and adaptation strategies. The prompts improve frozen-LLM robustness, mitigate fine-tuning degradation, and interact especially strongly with Whisper.

  • Frozen LLM: 2.45% Average F1 improvement (91.09% →93.54%) results from adding openSMILE to Whisper-backed Qwen-0.5B with a frozen LLM.ML F1 also rises 2.92% (90.50% →93.42%).
  • LoRA adaptation: 22.99% ML F1 improvement (68.18% →91.17%) shows that openSMILE makes LoRA-16 adaptation highly synergistic for the 0.5B Whisper-backed model.The result surpasses the no-openSMILE frozen baseline.
  • Native multimodal integration: Qwen2.5-Omni-3B receives only a marginal +1.28% Average F1 improvement from openSMILE injection.The passage attributes this potentially to its 12.5 Hz integrated encoder resolution versus 50 Hz for the evaluated encoders.
  • Encoder effects: 93.78% Average F1 is achieved by the 0.5B Whisper-backed LoRA-16 model with openSMILE, up from 86.48%, a gain of 7.30%.Whisper’s ASR pretraining suppresses non-linguistic micro-artifacts, leaving explicit prompts to supply forensic evidence.
  • Encoder effects: Wav2Vec2 gains are modest and complementary, with openSMILE adding 2.49% Average F1 for the 0.5B LoRA-16 model.Its self-supervised pretraining preserves low-level acoustic variation from the raw waveform.
  • Summary: openSMILE prompts bridge the modality gap and improve generalization, while mitigating degradation from LLM fine-tuning relative to frozen LLMs.The section evaluates prompt effects across encoder families and LLM adaptation strategies.

VII. COMPARISON WITH STATE OF THE ART SYSTEMS

The proposed textually grounded LLM configurations are compared with classical, self-supervised, and ALLM systems on unseen ITW and ML domains. They deliver strong out-of-domain generalization with lightweight frozen or LoRA-adapted configurations.

  • Evaluation setting: The comparison reports Macro-F1 on unseen ITW and ML because evaluated ALLM systems approach 100% F1 on in-domain LA19.Out-of-domain variation is therefore the focus of the comparison.
  • Enhancing SSL encoders: 79.80% F1 on ML is the highest reported result for cited traditional-classifier architectures using similar SSL encoders.The proposed Wav2Vec2-based configuration is described as superior in generalization.
  • Enhancing SSL encoders: 77.27% ITW and 91.84% ML are reached by Wav2Vec2 FT + Qwen-0.5B Frozen + OS.The configuration is presented as evidence that adding a lightweight frozen LLM to an acoustic encoder benefits generalization.
  • Standalone and ALLM strategies: 83.96% ITW and 93.42% ML are achieved by the frozen Whisper configuration with openSMILE, versus 71.11% and 67.49% for Whisper FT alone.The comparison highlights the benefit of combining the encoder with a lightweight LLM and explicit acoustic descriptors.
  • Standalone and ALLM strategies: 87.53% ITW and 91.17% ML are reached by the Whisper-based LoRA-16 framework with explicit openSMILE descriptions.The framework uses approximately 1.1B parameters in lightweight configurations.

VIII. CONCLUSION

The paper presents an ALLM framework that bridges acoustic physics and LLM semantics for deepfake voice detection. Frozen-LLM optimization supports robust generalization, while openSMILE grounding improves performance and works synergistically with LoRA adaptation.

  • VIII. CONCLUSION: Optimizing the audio encoder allows a frozen LLM to generalize robustly without overfitting.The conclusion frames this as a balance between computational efficiency and detection performance.
  • VIII. CONCLUSION: Structured openSMILE text tokens provide semantic context that enhances the frozen baseline and synergizes with LoRA adaptation.The framework is presented as bridging acoustic physics and LLM semantics.
  • VIII. CONCLUSION: Interpretable acoustic statistics are positioned as a basis for transparent, reasoning-based, explainable audio forensics.Future work will investigate prompt design and alternative cross-modal alignment methods.
Loading 2608.30622v1…