Source-linked AI summary
End-to-End Spectro-Temporal Graph Attention Networks for Speaker Verification Anti-Spoofing and Speech Deepfake Detection
Hemlata Tak, Jee-weon Jung, Jose Patino, Madhu Kamble, Massimiliano Todisco, Nicholas Evans
TL;DR
Spoofing cues occur across spectral sub-bands and temporal segments, while different attention mechanisms perform differently across diverse attacks. RawGAT-ST performs model-level spectro-temporal graph fusion directly from raw waveforms, with graph pooling, and achieves the best reported results on ASVspoof 2019 logical access.
Problem
Different attention mechanisms work better for different spoofing attacks, and neither spectral nor temporal attention alone performs as well across diverse attacks.
Method
RawGAT-ST learns relationships among spectral and temporal cues using a spectro-temporal graph attention network operating directly on raw waveforms, with graph pooling and fused spectral-temporal sub-graphs.
Results
RawGAT-ST achieves the best reported results by a substantial margin on the standard ASVspoof 2019 logical access database.
Takeaways & Limitations
Spectral and temporal attention are complementary, while graph pooling substantially improves performance within the unified anti-spoofing model.
Takeaways & Limitations
Learnable pooling ratios and weighted selective fusion remain future directions, with these ideas being pursued on the ASVspoof 2021 database.
Abstract
from arXiv · showhide
Artefacts that serve to distinguish bona fide speech from spoofed or deepfake speech are known to reside in specific subbands and temporal segments. Various approaches can be used to capture and model such artefacts, however, none works well across a spectrum of diverse spoofing attacks. Reliable detection then often depends upon the fusion of multiple detection systems, each tuned to detect different forms of attack. In this paper we show that better performance can be achieved when the fusion is performed within the model itself and when the representation is learned automatically from raw waveform inputs. The principal contribution is a spectro-temporal graph attention network (GAT) which learns the relationship between cues spanning different sub-bands and temporal intervals. Using a model-level graph fusion of spectral (S) and temporal (T) sub-graphs and a graph pooling strategy to improve discrimination, the proposed RawGAT-ST model achieves an equal error rate of 1.06 % for the ASVspoof 2019 logical access database. This is one of the best results reported to date and is reproducible using an open source implementation.
1. Introduction
Spoofing artefacts vary across spectral sub-bands and temporal segments, and separate attention models perform differently across diverse attacks. RawGAT-ST addresses this by learning spectro-temporal relationships and fusing complementary information within one end-to-end raw-waveform model.
- Spoofing cues occur in specific spectral sub-bands and temporal segments, motivating attention mechanisms that target both domains.
- Separate spectral and temporal attention models work better for different spoofing attacks and neither performs as well alone across diverse attacks.
- RawGAT-ST explores earlier, model-level fusion to jointly exploit complementary spectral and temporal information.
- The proposed architecture learns feature representations and graph attention directly from raw waveform inputs in a fully end-to-end system.
- Its spectro-temporal GAT learns relationships between cues at different sub-band and temporal intervals.
- A graph pooling strategy reduces graph dimension and improves discrimination, alongside exploration of model-level graph fusion strategies.
2. Related works
Prior work applied graph attention separately to spectral or temporal spoofing artefacts and combined their outputs through score-level fusion. This paper extends that direction toward a single model using model-level spectro-temporal fusion.
- Graph neural networks, including GATs, have been applied to speech processing tasks to model relationships among audio representations.
- The authors’ prior GAT-S and GAT-T systems used self-attention to emphasize informative spectral sub-bands or temporal intervals and their relationships.
- Those spectral and temporal models showed complementarity through score-level, or late, fusion.
- The proposed architecture contains separate spectral and temporal attention blocks followed by a spectro-temporal graph attention block, with model-level fusion.
- The present work hypothesizes that integrating the two models through early fusion can better leverage complementary information while using a single model.
3. Graph attention networks for anti-spoofing
A GAT represents feature-map elements as graph nodes and learns attention-weighted relationships among neighboring nodes. Each layer transforms node features, aggregates information, and produces an output graph.
- The graph is built from higher-level feature representations, with nodes representing spectral sub-bands or temporal frames and node dimensions matching feature-map channels.
- A GAT layer maps an input graph to an output graph by aggregating neighboring node features through learnable self-attention weights.
- The layer computes each output node from aggregated neighboring information and a projected residual connection, followed by batch normalization and SeLU activation.
- Attention weights encode the connective strength between node pairs, with larger weights indicating that one node is more informative of another.
- The attention weights are derived for neighboring nodes, including the node’s self-connection, using a learned mapping and element-wise feature interaction.
4. RawGAT-ST model for anti-spoofing and speech deepfake detection
RawGAT-ST learns representations from raw waveforms and models spectral, temporal, and cross-domain relationships with graph attention, pooling, and model-level fusion.
- RawGAT-ST comprises raw-waveform feature learning, spectro-temporal graph attention, discriminative graph pooling, and model-level fusion.The architecture is organized into four stages and ends with bona fide-versus-spoofed prediction.
- 4.1. Front-end (higher-level) feature representation: A sinc convolution front end and 2D residual network transform raw audio into higher-level time-frequency features S ∈ R^C×F×T.The residual blocks use batch normalization, SeLU activation, 2D convolution, and max-pooling for downsampling.
- 4.2. Spectro-temporal attention: Separate spectral and temporal GAT blocks identify cues in sub-bands and temporal intervals before a third block models relationships spanning both domains.The spectral and temporal blocks collapse the complementary dimension by max-pooling, then construct separate graphs for attention processing.
- 4.3. Graph pooling: Graph pooling selects informative nodes using learnable projection scores, retaining top-k nodes and sigmoid-weighting their features.The pooled spectral and temporal graphs are projected into a common dimensional space before fusion.
- 4.4. Model-level fusion: Model-level fusion combines the pooled spectral and temporal graphs through element-wise addition, multiplication, or concatenation, followed by another GAT and pooling stage.The resulting fused graph is processed into a final pooled representation for two-class prediction.
5. Experimental setup
Experiments use the ASVspoof 2019 logical access database to evaluate RawGAT-ST against end-to-end and domain-specific systems under varied spoofing attacks.
- Experiments use independent train, development, and evaluation partitions containing six training/development attacks and thirteen evaluation attacks.Spoofed speech is generated using speech synthesis and voice conversion algorithms.
- 5.2. Baseline: RawNet2 is the end-to-end baseline, using a sinc-filter front end followed by residual and recurrent layers to classify bona fide or spoofed speech.Input waveforms are truncated or concatenated into approximately four-second segments.
- 5.2. Baseline: Prior temporal and spectral GAT systems are excluded as baselines because they use hand-crafted features rather than end-to-end processing.End-to-end temporal and spectral variants are nevertheless reported using corresponding model sub-blocks.
- 5.3. RawGAT-ST implementation: The implementation reduces the first sinc layer to 70 filters, applies channel masking, uses smaller residual blocks, and adds empirically selected graph-pooling ratios.Channel masking randomly zeros contiguous sinc channels during training to improve generalization.
- 5.3. RawGAT-ST implementation: Training jointly optimizes the feature extractor and classifier with weighted cross entropy, Adam, mini-batches of 10, learning rate 0.0001, and 300 epochs.Bonafide and spoofed trials receive weights in a 9:1 ratio to manage training-set imbalance.
6. Experimental Results
RawGAT-ST substantially improves spoofing detection over RawNet2, with the strongest configuration benefiting from spectro-temporal attention and graph pooling. Ablations indicate that spectral attention and pooling are especially important, while comparisons place the system among strong single-model approaches.
- Results: 1.06% EER and 0.0335 min t-DCF are achieved by RawGAT-ST-mul, compared with 5.54% and 0.1547 for RawNet2.These are pooled results on the ASVspoof 2019 logical access evaluation database.
- Results: All RawGAT-ST configurations outperform the RawNet2 baseline by a substantial margin.The systems use different spectro-temporal fusion strategies.
- Ablation study: Removing spectral attention degrades performance by 34%, whereas removing temporal attention causes a smaller 13% relative degradation.The ablation results indicate greater importance for spectral attention, although both attention types are beneficial.
- Ablation study: Removing graph pooling produces a 58% relative performance degradation, showing its benefit for concentrating on informative node features.The pooled system scores 0.0335, compared with 0.0788 without pooling.
- Performance comparison: RawGAT-ST outperforms competing attention-based single systems and remains among the least complex systems, using 0.44M parameters.It also improves on the previous temporal GAT with late score fusion by 63% in min t-DCF and 77% in EER.
7. Conclusions
The paper presents model-level spectro-temporal graph attention for spoofing and speech deepfake detection, operating directly on raw waveforms. RawGAT-ST reports strong ASVspoof 2019 logical access results, while the authors identify further improvements for newer evaluation data and adaptive feature selection.
- Conclusions: RawGAT-ST uses model-level spectro-temporal attention to learn relationships among spoofing cues across sub-bands and temporal intervals.Its self-attention mechanism identifies discriminative nodes within the resulting graph.
- Conclusions: The model operates directly on raw waveforms without hand-crafted features and is among the least complex reported solutions.The comparison includes systems using raw and hand-crafted inputs.
- Conclusions: RawGAT-ST achieves the best reported results by a substantial margin on the ASVspoof 2019 logical access database.The conclusion also attributes benefits to spectral and temporal attention and graph pooling based on ablation results.
- Conclusions: A fully end-to-end approach is supported by comparisons between systems using raw signals and systems using hand-crafted inputs.The paper frames direct raw-signal operation as beneficial in this comparison.
- Future work: Future work targets learnable pooling ratios, weighted selective fusion, and evaluation on the ASVspoof 2021 challenge database.These directions are presented as having potential to improve performance further.