PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangled Global-Local Feature Learning with E-Branchformer for Audio Deepfake Detection

Phuong Tuan Dat, Ho Bao Thu, Nguyen Tran Trung, Pham Viet Hoang, Nguyen Thi Thu Trang

arXiv:2609.08948v1cs.SD

TL;DR

Voice synthesis advances threaten speech-based authentication, and effectively using high-dimensional self-supervised speech representations remains a deepfake-detection challenge. The paper proposes an E-Branchformer architecture that processes global and local features in parallel and refines the classification token. It reports state-of-the-art EERs across three benchmarks, while noting that its system-level comparison does not isolate the backbone’s contribution.

  • Problem

    Voice conversion and text-to-speech advances threaten voice-based authentication, while effective architectural use of high-dimensional self-supervised representations remains challenging.

    Source: §1 · p. 1

  • Method

    The proposed E-Branchformer deepfake detector uses parallel global-attention and local-convolution processing, with depthwise convolution and Squeeze-and-Excitation refining the class token.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    EERs of 0.88%, 1.85%, and 6.30% on ASVspoof 2021 LA, DF, and In-the-Wild establish state-of-the-art performance on all three datasets.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Ablations support complementary discriminative information from dual branches, improved SSL feature integration with Squeeze-and-Excitation Aggregation, and the importance of combining DWConv and SE for class-token enhancement.

    Source: §6 · p. 8

    The reported system-level contrast includes dual class tokens and DWConv–SE fusion, so it does not isolate the encoder’s contribution.

    Source: §4.4 · p. 6

Abstract

from arXiv · show

The rapid advancement of voice synthesis technologies such as text-to-speech and voice conversion poses significant threats to speech-based authentication systems, necessitating robust deepfake detection methods. In this work, we propose a novel E-Branchformer-based architecture that effectively leverages self-supervised speech representations for audio deepfake detection. Our model employs parallel branches to simultaneously capture global contextual dependencies through multi-head self-attention and local temporal patterns through convolutional processing. To enhance discriminative capability, we integrate depthwise convolution and Squeeze-and-Excitation modules that enrich the classification token with refined patch token information after feature merging. Extensive experiments on ASVspoof 2021 LA, DF, and In-the-Wild datasets demonstrate state-of-the-art performance with equal error rates of 0.88%, 1.85%, and 6.30% respectively, substantially outperforming existing methods. Comprehensive ablation studies validate that the dual-branch architecture provides complementary discriminative information, Squeeze-and-Excitation Aggregation significantly improves SSL feature integration, and the combination of DWConv and SE modules is critical for effective class token enhancement. The superior performance on real-world scenarios demonstrates strong generalization capability to diverse acoustic conditions and unseen spoofing attacks.

1 Introduction

Rapidly advancing voice synthesis threatens voice-based authentication, while exploiting high-dimensional self-supervised speech representations remains an architectural challenge. This work applies E-Branchformer to audio deepfake detection and reports state-of-the-art results across three benchmarks.

1 Introduction

  • Voice conversion and text-to-speech advances threaten the integrity of voice-based authentication systems.Speech foundation-model representations are proposed as a way to improve anti-spoofing robustness and generalization.Source: §1 · p. 1
  • Processing high-dimensional self-supervised representations effectively remains a key architectural challenge for deepfake detection.Convolution- and Transformer-based downstream models have been used, and Conformer combines local and global context sequentially.Source: §1 · p. 1
  • The proposed design disentangles global and local feature learning with dual class tokens and fuses their representations using depthwise convolution and Squeeze-and-Excitation modules.The paper also investigates local-global merging strategies and analyzes feature contributions through ablations.Source: §1 · p. 2
  • EERs of 0.88%, 1.85%, and 6.30% on ASVspoof 2021 LA, DF, and In-the-Wild establish state-of-the-art results across all three datasets.The proposed system is the first application of E-Branchformer to audio deepfake detection.Source: §1 · p. 2

2 Preliminaries

Conformer processes its components sequentially, whereas E-Branchformer places global self-attention and local convolutional-gating branches in parallel. E-Branchformer merges branch outputs using concatenation, depthwise convolution, and linear projection.

2.1 Conformer

  • Conformer arranges two feed-forward networks, multi-head self-attention, and convolution sequentially, with residual connections and pre-normalization.Its convolution module includes pointwise and depthwise convolutions, GLU and Swish activations, and batch normalization.Source: §2.1 · p. 2

2.2 E-Branchformer

  • E-Branchformer uses parallel branches: multi-head self-attention models global, long-range context, while convolutional gating MLP captures local, short-range patterns.The architecture retains the dual Macaron-style feed-forward configuration.Source: §2.2 · p. 2
  • E-Branchformer merges its branch outputs with concatenation, 1-D depthwise convolution, and linear projection, a strategy reported as more effective than straightforward concatenation and projection.The comparison is with the original Branchformer architecture.Source: §2.2 · p. 2

3 Proposed Architecture

The baseline extracts XLS-R speech representations, projects them, and uses a prepended classification token with a Conformer encoder. The proposed architecture aggregates XLS-R layers and processes features through parallel global-attention and local-convolution branches, then refines the class token using patch-token information.

3.1 Baseline Model

  • The XLSR-Conformer baseline projects pretrained XLS-R features and prepends a learnable classification token whose final encoder state determines whether speech is bonafide or spoofed.The Conformer encoder stacks blocks with self-attention and convolution between feed-forward modules.Source: §3.1 · pp. 2–3

3.2 Overall Proposed Model

  • The proposed model aggregates representations across XLS-R layers, projects them, and concatenates them with a learnable classification token.The aggregation module can use strategies including weighted summation, Squeeze-and-Excitation Aggregation, and Attentive Merging.Source: §3.2 · p. 3
  • Stacked E-Branchformer blocks process speech with parallel self-attention for global dependencies and convolution for local temporal patterns.The architecture is designed to preserve the complementary strengths of global modeling and local feature extraction.Source: §3.2 · pp. 3–4
  • After branch fusion, patch tokens pass through depthwise convolution and Squeeze-and-Excitation to produce channel weights applied to the class token before classification.The resulting enriched class token and recalibrated patch tokens feed the classification stage.Source: §3.2 · p. 4

4 Experiments

Experiments compare the proposed model with recent systems across ASVspoof 2021 and In-the-Wild benchmarks, and also evaluate it on ASVspoof 5. The reported EERs set new benchmarks on the three ASVspoof 2021 and In-the-Wild evaluations.

4.1 Datasets and Evaluation Metrics

  • The evaluation covers controlled ASVspoof datasets, including codec-processed DF21, and 31,779 real-world ITW test samples collected across varied acoustic conditions and recording devices.Source: §4.1 · p. 4

4.2 Experimental Setup

  • The XLS-R 300M front end is jointly fine-tuned, with its 1024-dimensional output projected to 144 dimensions before a four-block E-Branchformer encoder.Source: §4.2 · p. 5

4.3 Overview of State-of-the-Art Models

  • Table 3 compares the proposed model with recent state-of-the-art systems on the ASVspoof 2021 evaluation sets and ITW.Source: Table 3 · p. 5, §4.4 · p. 5

4.4 Experimental Results

  • EERs of 0.88% on ASVspoof 2021 LA, 1.85% on ASVspoof 2021 DF, and 6.30% on ITW establish new benchmarks across all three evaluations.Source: §4.4 · p. 5, Table 3 · p. 5
  • The reported XLSR-Conformer comparisons are between complete systems, including dual class tokens and DWConv–SE fusion, rather than isolating the encoder alone.Source: §4.4 · p. 6
  • On ASVspoof 5, the model achieves 5.44% EER, a relative reduction of 8.1% versus the strongest baseline, Nes2Net-X, at 5.92%.Source: §4.4 · p. 6, Table 4 · p. 6

5 Ablation Study

The ablations find complementary contributions from global and local branches, adaptive SSL-layer aggregation, and DWConv–SE fusion. Layer-wise attention analysis distinguishes E-Branchformer’s broader attention patterns from Conformer’s more local ones.

5.1 Impact of Global and Local Branches

  • 0.94% to 0.88% on 21LA, 2.29% to 1.85% on 21DF, 6.73% to 6.30% on ITW, and 6.16% to 5.44% on ASVspoof 5 when combining branches versus the better single branch.The global branch alone consistently outperforms the local branch, while the local branch contributes complementary cues.Source: §5.1 · p. 7

5.2 Impact of SSL Feature Aggregation Strategies

  • SEA achieves EERs of 0.88% on 21LA, 1.85% on 21DF, 6.30% on ITW, and 5.44% on ASVspoof 5, outperforming AttM, weighted sum, and last-hidden-layer aggregation.The authors report that adaptive, content-aware SSL-layer selection is critical for extracting informative representations.Source: §5.2 · p. 7

5.3 Impact of Key Architectural Components

  • Removing both DWConv and SENet yields the worst results: 1.06% EER on 21LA, 2.87% on 21DF, 10.90% on ITW, and 7.18% on ASVspoof 5.Removing either component degrades performance across benchmarks; DWConv contributes more, particularly on 21DF and ITW.Source: §5.3 · p. 7, Table 6 · p. 7

5.4 Layer-Wise Analysis of Global Branch

  • ∼0.510–0.522 on LA and ∼0.511–0.525 on DF for E-Branchformer versus ∼0.556–0.558 and ∼0.556 for Conformer, with non-overlapping score ranges across layers and sets.The comparison uses identical XLS-R inputs and training protocols; the authors interpret the pattern as local extraction being delegated to E-Branchformer’s convolutional branch, leaving attention to model broader dependencies.Source: §5.4 · pp. 7–8, Figure 3 · p. 8

6 Conclusion

The conclusion presents an E-Branchformer architecture for audio deepfake detection that combines global and local processing of self-supervised speech representations. It reports state-of-the-art results on three benchmarks and identifies component contributions and future research directions.

  • The proposed E-Branchformer architecture processes self-supervised speech representations in parallel to capture global and local features for audio deepfake detection.Source: §6 · p. 8
  • The model achieves state-of-the-art EERs of 0.88% on ASVspoof 2021 LA, 1.85% on DF, and 6.30% on In-the-Wild.Source: §6 · p. 8
  • Ablations support complementary information from attention and convolution branches, better SSL feature integration with SEA, and a critical role for combined DWConv and SE in class-token enhancement.Source: §6 · p. 8
  • The authors identify real-world acoustic diversity and unseen spoofing attacks as the scope of the reported generalization, and propose multimodal detection and adversarial robustness for future work.Source: §6 · p. 8
Loading 2609.08948v1…