Source-linked AI summary
Tracing Decoder Artifacts for Compact Synthetic Speech Screening
Yi Chen Liu, Jian Liu
TL;DR
Large pretrained detectors can be costly to run on every recording, motivating a compact front-end screen that routes suspicious inputs for more expensive analysis. The paper measures decoder-related spectral traces alongside spectro-temporal features; across seven generators and two human-speech sources, it reports 0.021% EER, and a simulated cascade reduces estimated energy by 84.4% at a 0.050% miss rate.
Problem
High-performing synthetic-speech detectors increasingly rely on large pretrained encoders, which can impose substantial computational and energy costs when applied to every recording.
Source: §1 · p. 1Method
The screen measures spectral artifacts associated with decoder operations and combines them with spectro-temporal descriptors in a compact gradient-boosted tree.
Source: §1 · p. 1Results
0.021% EER across seven generators and two human-speech sources accompanies 151 KiB estimated storage; the simulated cascade reduces estimated energy by 84.4% at a 0.050% synthetic-speech miss rate.
Source: §1 · p. 1Takeaways & Limitations
Decoder-guided acoustic measurements can provide a compact first-stage screen that avoids unnecessary large-model inference while retaining interpretable connections to speech generation.
Source: §1 · p. 1Cross-generator transfer remains the main challenge, and held-out-generator EER varies from 0.014% to 6.747%.
Source: §5 · p. 4, §4.2 · p. 4
Abstract
from arXiv · showhide
Recent advances in speech synthesis and voice cloning have increased the need for reliable synthetic-speech detection, yet high-accuracy detectors increasingly rely on large pretrained models that are costly to invoke on every recording. Rather than replacing such detectors, we investigate a compact front-end screen that processes all inputs cheaply and forwards only suspicious recordings for more expensive analysis. To enable lightweight screening without a large learned encoder, we exploit spectral traces introduced by speech-generation operations. We analyze how learned upsampling and inverse short-time Fourier transform synthesis can produce predictable spectral artifacts and measure their presence directly in generated waveforms. Because the strength of these artifacts varies across generators, we combine decoder-guided spectral measurements with complementary descriptors of short-time spectral shape and temporal variation in a compact gradient-boosted tree. Across seven speech generators and two human-speech sources, the proposed screen achieves an equal error rate of 0.021\% with an estimated model storage of 151 KiB. When used as the first stage of a simulated cascade with a 1.15-billion-parameter detector, it reduces estimated detection energy by 84.4\% while operating at a 0.050\% synthetic-speech miss rate, demonstrating the potential of decoder-guided acoustic evidence for low-cost front-end screening.
1. INTRODUCTION
Large pretrained detectors can provide strong synthetic-speech detection, but applying them to every recording is costly. This paper proposes a compact front-end screen based on generation traces and complementary acoustic features, achieving low error and reducing estimated cascade energy.
1. INTRODUCTION
- The proposed screen processes each recording locally and forwards only suspicious inputs to a more expensive backend, aiming to reduce unnecessary inference and remote audio transmission.It must remain inexpensive per input while keeping the synthetic-speech miss rate sufficiently low for benign recordings to bypass downstream processing.Source: §1 · p. 1
- Learned upsampling and inverse short-time Fourier transform synthesis can leave structured spectral traces that are measurable directly from generated waveforms.Prior work reports that vocoder-resynthesized speech can help detect speech produced by complete text-to-speech and voice-conversion systems, and periodic synthesis operations can leave spectral traces.Source: §1 · p. 1
- Because artifact strength varies across generators, the screen combines decoder-guided measurements with LFCC descriptors of short-time spectral shape and temporal variation in a compact gradient-boosted tree.The artifact measurements include local prominence and persistent spectral residuals [4], while the temporal descriptors follow prior work.Source: §1 · p. 1
- 0.021% EER and 151 KiB storage characterize the screen across seven generators and two human-speech sources; a simulated cascade reduces estimated energy by 84.4% at a 0.050% synthetic-speech miss rate.The screen is intended to forward suspicious recordings to a more expensive backend rather than replace it.Source: §1 · p. 1
2. RELATED WORK
Prior work expands synthetic-speech datasets and studies both large-encoder and lightweight detection approaches. It also documents variation across sources and recording conditions, motivating cross-domain evaluation.
2. RELATED WORK
- Recent datasets cover more speakers, acoustic conditions, adversarial attacks, languages, and synthesis methods, including text-to-speech, voice conversion, and neural vocoder outputs [6].Evaluations across datasets show substantial performance variation across sources and recording conditions.Source: §2 · pp. 1–2
- Self-supervised detectors use speech-encoder representations, while lightweight approaches use compact networks or explicit acoustic features, including waveform, cepstral, and temporal-difference features.Source: §2 · p. 2
- Related artifact-based approaches learn frequency-domain fingerprints [12] or extract residual fingerprints for detection and model attribution [13].Source: §2 · p. 2
3. ARTIFACT-GUIDED DETECTION
The detector combines decoder-guided spectral evidence with spectro-temporal descriptors in a compact gradient-boosted tree. It derives artifact locations from synthesis operations, measures their waveform evidence, and supplements them with features of spectral shape and temporal variation.
3.1.1. Artifact Origins and Frequencies
- Unequal phase sums in transposed-convolution filters can turn a nonzero decoder-feature mean into a period-stride response, while iSTFT synthesis can also yield periodic spectral components.The periodic transposed-convolution response is described as a one-dimensional checkerboard mechanism; the specific traces depend on synthesis operations and learned filters.Source: §3.1.1 · p. 2
- Layer spectra and waveform measurements show predicted artifact locations, including peaks near 6 and 10 kHz in 200 CosyVoice2 recordings.The example analysis also identifies a possible 5512.5-Hz DC replica in OpenVoice and possible components at 6 and 10 kHz in CosyVoice2.Source: §3.1.1 · p. 2, Figure 1 · p. 2
3.1.2. Waveform Measurements
- The waveform measurements combine normalized broadband spectral residuals with prominence measurements at 23 candidate decoder-guided frequencies, where peaks are compared with nearby spectral background.Candidate frequencies indicate where evidence is measured; they do not assume every generator has a strong peak there.Source: §3.1.2 · pp. 2–3
3.2. Spectro-Temporal Features
- LFCC trajectories are summarized by distributional statistics and temporal differences, while modulation features capture spectral-shape fluctuation strength across temporal rates.The features use 40 LFCCs; modulation magnitudes are averaged across eight logarithmically spaced bands spanning 1–50 Hz.Source: §3.2 · p. 3, Figure 2 · p. 3
3.3. Feature Fusion & Classification
- The classifier concatenates broadband residuals, decoder-guided peak measurements, LFCC statistics, and temporal modulation features in a histogram-based gradient-boosted tree.Training uses class- and source-balanced weights, and the test set is reserved for final evaluation.Source: §3 · p. 2, §3.3 · p. 3
4. EXPERIMENTS
Across seen sources, the compact detector matches the best neural comparator, while unseen-generator transfer varies substantially. In a simulated cascade, screening reduces estimated detection energy at a stringent synthetic-speech miss rate.
4.1. Evaluation Protocols
- Evaluation includes seen-source, cross-source, and leave-one-generator-out settings, with disjoint train, validation, and test recordings in the seen-source evaluation.The cross-source evaluation uses fixed thresholds without retraining, and preprocessing uses 16-kHz mono audio with the leading four seconds and no augmentation, loudness normalization, silence trimming, or additional lossy encoding.Source: §4.1 · pp. 3–4
4.2. Detection Performance and Generalization
- 0.021% seen-source EER matches AASIST, with 151 KiB of tree storage versus 694 KiB–29.2 MiB for the neural models.The evaluation uses recordings from seven synthetic-speech generators and two human-speech sources.Source: §4.2 · p. 4, Table 1 · p. 4
- 82.5–99.5% recall on unseen synthetic speech varies by generator, while 98.4% of unseen LibriSpeech recordings are correctly accepted as human.Neural comparators reach 94–100% recall across the four unseen synthetic sources; the authors identify generator shift as the more important limitation.Source: §4.2 · p. 4
- LOGO EER ranges from 0.014% for CosyVoice held out to 6.747% for MeloTTS held out, averaging 2.893%.The wide range indicates that cues from the remaining generators transfer strongly to some held-out systems but not others.Source: §4.2 · p. 4, Table 2 · p. 4
4.3. Screening Efficiency
- 84.4% lower estimated energy reduces the 2.981 kWh direct-detection estimate to 0.466 kWh for 10^6 requests.At a 0.050% miss-rate operating point, about 90% of requests bypass the backend under a workload with 10% synthetic-speech prevalence; the estimate assumes constant backend energy and excludes model loading, warmup, GPU idle energy, and backend host costs.Source: §4.3 · p. 4, Table 3 · p. 4
5. CONCLUSION
The paper presents a compact synthetic-speech screen combining decoder-guided artifacts and spectro-temporal features. Seen-source performance is strong, but cross-generator transfer remains the main challenge.
5. CONCLUSION
- The screen combines decoder-guided spectral artifacts with complementary spectro-temporal features.It achieves 0.021% seen-source EER with 151 KiB estimated model storage; the screening analysis indicates potential to reduce expensive backend inference.Source: §5 · p. 4