Source-linked AI summary
EnvSDD: Benchmarking Environmental Sound Deepfake Detection
Han Yin, Yang Xiao, Rohan Kumar Das, Jisheng Bai, Haohe Liu, Wenwu Wang, Mark D Plumbley
TL;DR
Environmental sound deepfake detection needs dedicated datasets and methods amid increasingly realistic AI-generated sounds and potential misuse. EnvSDD pairs a large-scale dataset with a BEATs-based detector, which outperforms the baselines across evaluation conditions.
Problem
Environmental sounds may require detection methods distinct from speech and singing, while realistic AI-generated sounds create potential for misleading fake audio.
Source: §1 · p. 1Method
EnvSDD provides 45.25 hours of real and 316.74 hours of fake environmental audio, alongside a detector that adds pre-trained BEATs to AASIST.
Source: §1 · p. 1, §4.2 · p. 4Results
BEATs+AASIST outperforms both baseline models across all test conditions.
Source: §1 · p. 1, §4.2 · p. 4Takeaways & Limitations
The dataset, models, and codes are publicly available.
Source: §1 · p. 1The proposed system's generalization to unseen domains remains limited.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Audio generation systems now create very realistic soundscapes that can enhance media production, but also pose potential risks. Several studies have examined deepfakes in speech or singing voice. However, environmental sounds have different characteristics, which may make methods for detecting speech and singing deepfakes less effective for real-world sounds. In addition, existing datasets for environmental sound deepfake detection are limited in scale and audio types. To address this gap, we introduce EnvSDD, the first large-scale curated dataset designed for this task, consisting of 45.25 hours of real and 316.74 hours of fake audio. The test set includes diverse conditions to evaluate the generalizability, such as unseen generation models and unseen datasets. We also propose an audio deepfake detection system, based on a pre-trained audio foundation model. Results on EnvSDD show that our proposed system outperforms the state-of-the-art systems from speech and singing domains.
1. Introduction
Environmental sound deepfake detection addresses the risks of realistic AI-generated sounds and the distinct acoustic challenges that separate environmental audio from speech and singing. The paper introduces EnvSDD and a detection system that outperforms the baselines across evaluation scenarios.
1. Introduction
- Realistic AI-generated environmental sounds may support media production but also pose risks of misinformation and misleading fake audio.Source: §1 · p. 1
- Environmental sounds can contain simultaneous events and lack the steady rhythms and fixed tones of speech or singing, raising questions about direct transfer of those detection methods.Source: §1 · p. 1
- Existing environmental-sound datasets address scene consistency or have limits in audio type, scale, or evaluation of generalization to unseen domains.SceneFake and EnvFake combine real-life sounds, while the cited Foley and inpainting datasets have monophonic or polyphonic scope and evaluation or scale constraints.Source: §1 · p. 1
- The proposed system outperforms the baseline systems across all evaluation scenarios.The paper introduces EnvSDD, with 45.25 hours of real audio and 316.74 hours of deepfake environmental audio.Source: §1 · p. 1
2. EnvSDD Dataset
EnvSDD samples real-life environmental audio and generates deepfake clips using text-to-audio and audio-to-audio models. It includes monophonic and polyphonic data, and defines evaluation splits for in-domain and out-of-domain testing.
2. EnvSDD Dataset
- EnvSDD combines sampled real-life audio with deepfake clips generated by text-to-audio and audio-to-audio models.Source: §2 · p. 2
2.1. Real Data Collection
- The real-data collection covers monophonic clips from UrbanSound8K and DCASE 2023 Task7 Dev, and polyphonic clips from TAU UAS 2019, TUT SED 2016/2017, and Clotho.Clips are segmented into four-second samples and resampled to 16 kHz; DCASE 2023 Task7 Dev and Clotho are reserved for evaluation, with overlapping DogBark and GunShot events excluded.Source: §2.1 · p. 2
2.2. Deepfake Data Generation
- Deepfake generation uses five text-to-audio models and two audio-to-audio models, including AudioLDM and AudioLDM 2 for audio-to-audio generation.AudioLDM conditions on CLAP features, whereas AudioLDM 2 uses AudioMAE features and produces outputs that more closely match the original acoustically.Source: §2.2 · pp. 2–3
- For monophonic clips, metadata is rewritten as the caption; for polyphonic clips, Mistral 7B generates captions from metadata, except for Clotho, which already has captions.Source: §2.2 · p. 2
2.3. Dataset Splits
- The dataset defines four test conditions: in-domain evaluation, unseen generation models, unseen source datasets, and both unseen models and datasets.Test 04 is designated the most challenging condition; D1–D6 and G1–G5 label source datasets and generation models.Source: §2.3 · p. 3
3. Model Architectures
The paper evaluates speech-domain detectors as baselines for environmental sound deepfake detection and proposes BEATs+AASIST, which replaces the speech-pretrained front end with an audio foundation model.
3.1. Baselines: AASIST and W2V2+AASIST
- AASIST learns spectral and temporal features with a heterogeneous stacking graph attention layer, while W2V2+AASIST uses wav2vec 2.0 XLS-R to extract waveform representations.W2V2 was pretrained on 128 languages and approximately 436K hours of unlabeled speech.Source: §3.1 · p. 3
3.2. Proposed BEATs+AASIST System for ESDD
- BEATs+AASIST feeds BEATs representations through a RawNet2-based encoder and self-attention aggregation layer into AASIST for two-class prediction.BEATs is pretrained on AudioSet-2M; the system replaces W2V2 as the front end.Source: §3.2 · p. 3
- The authors select BEATs because its pretraining on large-scale, diverse sound datasets is intended to capture environmental-audio features.The paper hypothesizes that a speech-pretrained model may generalize poorly to diverse environmental soundscapes.Source: §3.2 · p. 3
4. Experiments and Results
The experiments compare baseline and proposed systems using EER on EnvSDD, including seen and unseen test conditions and monophonic and polyphonic audio. BEATs+AASIST outperforms both baselines across test conditions, although performance degrades on unseen conditions.
4.2. Performance of Baselines and Proposed System
- BEATs+AASIST outperforms both baseline systems across all EnvSDD test conditions.Table 4 reports EER (%) for baseline and proposed systems; “Seen SD” and “Seen GM” denote seen source datasets and generation models.Source: Table 4 · p. 4, §4.2 · p. 4
4.1. Experimental Setup
- EER is the error rate at the threshold where false acceptance and false rejection rates are equal; lower EER indicates better detection.The experiments use a batch size of 32, Adam with weight decay 0.0001, and a maximum of 50 epochs, with early stopping after five epochs without validation-loss improvement.Source: §4.1 · pp. 3–4
4.2. Performance of Baselines and Proposed System
- All systems perform well on seen Test 01, but performance drops on unseen conditions; unseen generation models and source datasets each degrade results, and Test 04 combines both shifts.AASIST and BEATs+AASIST are more affected by unseen source datasets than unseen generation models, whereas W2V2+AASIST shows the opposite trend.Source: §4.2 · p. 4
4.3. Performance for Monophonic and Polyphonic Audio
- BEATs+AASIST generally has higher EER on monophonic than polyphonic audio, with an exception for TTA detection in Test 03; it beats W2V2+AASIST in all cases.W2V2+AASIST performs better on polyphonic audio in Tests 01–02, but better on monophonic audio in Tests 03–04.Source: §4.3 · p. 4, Figure 2 · p. 4
5. Conclusions
The paper presents EnvSDD and a BEATs-based detector for environmental sound deepfakes. The detector outperforms the speech- and singing-domain baselines on EnvSDD, but generalization to unseen domains remains limited.
5. Conclusions
- EnvSDD is presented as the first large-scale dataset of real and deepfake environmental audio, and the proposed BEATs-based system outperforms two speech- and singing-domain baselines on its test conditions.The authors identify limited generalization to unseen domains as a direction for future work.Source: §5 · p. 4