Source-linked AI summary
Input Representation, Not Model Size or Pretraining Scale, Shapes Speaker Verification Robustness to Temporal Degradation
Erdem Baha Topbas, Srikanth Madikeri, Elisa Pellegrino, Volker Dellwo
TL;DR
Whether speaker identity survives heavy temporal degradation that listeners can tolerate remains largely unexplored. The paper evaluates six ASV models trained on intact speech under three temporal degradations and finds that filterbank-input models degrade less than waveform-input counterparts in every family-by-condition comparison.
Problem
Whether speaker-specific information survives temporal degradation that leaves speech intelligible to listeners remains largely unexplored.
Source: §1 · p. 1Method
Six ASV models trained on intact speech were evaluated on VoxCeleb1-O under three degradations that retain different combinations of temporal cues.
Source: §1 · p. 1, §2.2 · p. 2Results
In all nine family-by-condition comparisons, filterbank-input models degraded less; in the controlled convolutional pair, PER-condition EER was 17.1% versus 29.7% for waveform input.
Source: §1 · p. 1Takeaways & Limitations
Under these conditions, restoring temporal fine structure recovered substantially more speaker information than restoring the envelope.
Source: §3 · p. 3, §5 · p. 4Because no canonical method removes periodicity while preserving other components, the evaluation covers only periodicity-retaining conditions.
Source: §2.1 · p. 2
Abstract
from arXiv · showhide
Speech remains intelligible to listeners under heavy temporal degradation, but whether speaker identity survives is largely unexplored. We evaluate six automatic speaker verification (ASV) models, trained on temporally intact speech, and tested under three degradations that remove different combinations of temporal cues. The models form three pairs, one per architecture family (convolutional, transformer, conformer), each pairing a waveform-input with a mel filterbank-input model. All models degrade, but robustness tracks input representation rather than pretraining scale or model size. In all nine family-by-condition comparisons the filterbank-input model degrades less, with eight of nine 95% confidence intervals excluding zero, even when it is smaller or has less pretraining than its waveform counterpart. In the controlled convolutional pair, where the models differ only in input representation, the filterbank-input model's equal error rate rises from 1.1% to 17.1% under the heaviest tested degradation, against 2.9% to29.7% for its waveform counterpart.
1. INTRODUCTION
Speech intelligibility can persist under temporal degradation, but whether speaker-specific information survives remains largely unexplored. Across six ASV models, filterbank-input models degraded less than waveform-input counterparts in all nine family-by-condition comparisons.
1. INTRODUCTION
- Speech temporal information comprises envelope (2–50 Hz), periodicity (50–500 Hz), and temporal fine structure (500 Hz–10 kHz).These components capture progressively faster waveform fluctuations.Source: §1 · p. 1
- Speech can remain comprehensible when a temporal component is removed, including under infinite peak clipping that retains periodicity alone.The cited listener accuracy under IPC is 50%–90%.Source: §1 · p. 1
- In the controlled convolutional pair, PER-condition EER was 17.1% for the filterbank-input model versus 29.7% for its waveform-input counterpart.Across all nine family-by-condition comparisons, filterbank-input models degraded less.Source: §1 · p. 1
- The study tests six ASV models across waveform and mel-filterbank inputs and three architecture families under three intelligible-to-humans temporal degradations.The tested conditions retain periodicity alone, periodicity plus envelope, or periodicity plus fine structure.Source: §1 · p. 1
2. METHODOLOGY
The study creates three periodicity-retaining degradations from broadband waveforms, trains and evaluates six ASV models on specified corpora, and compares performance using EER and minDCF. Its evaluation normalizes EER degradation by baseline headroom and reports paired bootstrap intervals for individual contrasts.
2.1. Temporal Degradation Techniques
- PER uses IPC; PER+ENV applies a smoothed Hilbert envelope to IPC, while PER+TFS divides out the envelope to retain waveform oscillations.The broadband decomposition differs from narrow-band methods, so reported magnitudes are not directly comparable with studies primarily using narrow-band components.Source: §2.1 · pp. 1–2
- No canonical method removes periodicity while preserving the other components, so the evaluation includes only periodicity-retaining conditions.Source: §2.1 · p. 2
2.2. Data and Corpora
- Models were trained on VoxCeleb2 using 2 s crops and noise and reverberation augmentation, then evaluated on condition-matched VoxCeleb1-O trial pairs.Source: §2.2 · p. 2
- The study selected four pretrained models and trained two ResNet models from scratch; all six produced 256-dimensional embeddings using temporal statistics pooling.The pretrained models were adapted with LoRA; common training used AAM loss.Source: §2.2 · p. 2
2.3. Model Evaluation
- Evaluation reports EER and minDCF, with EER degradation normalized as a percentage of the distance from baseline to chance performance.EER confidence intervals use speaker-level paired bootstrap; contrasts are assessed separately without multiplicity adjustment.Source: §2.3 · p. 2
3. RESULTS
Checkpoints trained on unmodified speech were evaluated under temporal degradation. Across all nine family-by-condition contrasts, waveform-input models degrade more than their filterbank-input counterparts, though one contrast is not statistically distinguishable from zero.
3. RESULTS
- The results are not directly comparable to [20] because the training conditions differ.Source: §3 · p. 2
- Under decomposition, most systems approach minDCF's 1.0 bound, making minDCF substantially more compressed than EER in this regime.The authors therefore base the input-representation contrast on EER.Source: §3 · p. 2
- All six models degrade under every condition; PER causes the largest drop, with some models nearing 40% EER, while chance-level ceiling effects make PER contrasts conservative.The controlled convolutional comparison is least affected by the ceiling effect.Source: §3 · pp. 2–3
- Across all nine family-by-condition contrasts, waveform-input models degrade more than filterbank-input counterparts, though one contrast is not statistically distinguishable from zero.The comparison is shown by architecture, condition, and input representation in Fig. 2; Table 1 also reports the paired models.Source: §3 · p. 3, Figure 2 · p. 3, Table 1 · p. 3
- Restoring TFS yields a mean 10.94-percentage-point gain, versus 1.32 percentage points for ENV, and TFS restoration exceeds ENV restoration in every model.PER+TFS is consistently the least damaging condition.Source: §3 · p. 3
- Speaker-level confidence intervals are a median 5.97× wider than trial-level intervals computed on the same scores.Trial-level resampling substantially overstates precision.Source: §3 · p. 3
4. DISCUSSION
Across architecture families and temporal-degradation conditions, filterbank-input models degrade less than waveform-input models; the discussion considers why this pattern is unlikely to reflect pretraining, model size, frame rate, or IPC noise artifacts alone.
4. DISCUSSION
- Mel filterbanks may be more robust because their transforms discard sample-level details perturbed by temporal degradation, allowing speaker-identifying characteristics to persist.The passage connects this explanation to mel filterbanks emphasizing qualities humans listen to and human speech recognition under similar degradation.Source: §4 · p. 3
- In every family-by-condition comparison, waveform-input models degrade more; only one of nine confidence intervals includes zero.Fig. 3 plots headroom-normalized degradation, and Table 2 groups degradation relative to unmodified speech by architecture family.Source: Figure 3 · p. 4, Table 2 · p. 4
- Pretraining and model size do not explain the pattern: Speech2Text is more robust than more-pretrained WavLM, and pretrained filterbank models are smaller than their waveform counterparts.The convolutional models also differ in sampling density, but the same pattern in the other pairs means frame rate alone cannot account for the effect.Source: §4 · pp. 3–4
- An IPC noise-lift explanation is inconsistent with the smallest contrast occurring for PER+TFS rather than PER+ENV in the controlled pair and every family.If the noise lift were responsible, PER+ENV would have the smallest contrast; the reported controlled-pair values are −21.53 for PER+ENV and −16.85 for PER+TFS.Source: §4 · p. 4
5. CONCLUSION
The study tests six ASV models under heavy temporal degradation and finds greater degradation in waveform-input than filterbank-input models. Restoring temporal fine structure recovers more speaker information than restoring the envelope.
5. CONCLUSION
- Six ASV models across three architecture families and two input representations were evaluated on temporally degraded speech.Every system degraded under all three conditions.Source: §5 · p. 4
- Within every model pair, waveform-input performance degraded more than filterbank-input performance, including in the controlled convolutional comparison.The conclusion reports that all intervals exclude zero for the controlled pair and that the pattern replicates in both pretrained pairs.Source: §5 · p. 4
- Restoring temporal fine structure recovered substantially more speaker information than restoring the envelope.The conclusion also states that both TFS and ENV carry substantial information in speaker recognition.Source: §5 · p. 4
7. FUNDING ACKNOWLEDGMENTS
The project received Swiss federal funding and was conducted as part of the VoCS project, which received European Union Horizon Europe funding.
7. FUNDING ACKNOWLEDGMENTS
- The project was funded by Switzerland’s State Secretariat for Education, Research and Innovation.The passage identifies it as the federal government’s secretariat.Source: §7 · p. 5
- The work was conducted as part of the VoCS project.Source: §7 · p. 5
- VoCS received funding through the European Union’s Horizon Europe Framework program under a Marie Skłodowska-Curie grant agreement.The grant agreement number is 101168998.Source: §7 · p. 5
8. COMPLIANCE WITH ETHICAL STANDARDS
The study used open-access VoxCeleb human-subject data, and ethical approval was not required according to the data licensing.