PaperlayerView exact revision on arXiv

Source-linked AI summary

ReDimNet2+: Multi-Corpus Data Scaling for Robust Speaker Verification

Kirill Borodin, Vasilii Kudryavtsev, Maxim Maslov, Grach Mkrtchian

arXiv:2609.37014v1cs.SD

TL;DR

Automatic speaker verification can degrade under unseen channels, compression, and other mismatched conditions. ReDimNet2+ combines multi-corpus training, waveform and codec augmentation, LMFT, and graph-based retrieval reranking. Under a shared 4-second protocol, it lowers pooled and degraded-condition VoxCeleb1 error rates relative to the starting checkpoint.

  • Problem

    ASV systems can degrade under unseen channels, aggressive compression, short fragments, or retrieval-style open-set evaluation, and diverse public training data alone does not ensure robustness to absent degradation mechanisms.

    Source: §1 · p. 1

  • Method

    ReDimNet2+ adapts a compact ReDimNet2 backbone using seven-corpus training, waveform and codec augmentation, staged LMFT, and graph-based retrieval reranking.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 2

  • Results

    Pooled VoxCeleb1 EER falls from 2.42% to 0.824%, and EERph falls from 7.214% to 1.991% under the shared 4-second protocol.

    Source: §4.2 · p. 3

  • Takeaways & Limitations

    The development sequence improves verification under the shared 4-second protocol, while reranking improves retrieval on VoxCeleb1 and a VoxBlink2 subset.

    Source: §5 · p. 4

    The development sequence changes several training factors together, external checkpoints are not controlled for training data, and synthetic EERph or sub-1% VoxCeleb1 EER does not establish in-the-wild robustness.

    Source: §5 · p. 4

Abstract

from arXiv · show

Automatic speaker verification must remain reliable across devices, rooms, and compression pipelines. We present ReDimNet2+, which scales training of the compact ReDimNet2 backbone across seven public corpora (63,934 speakers, about 8,675 hours). Analysis of a VoxBlink2 subset reveals a shift in predicted spectral coloration, motivating codec and waveform augmentation alongside this multi-corpus training, large-margin fine-tuning (LMFT), and graph-based retrieval reranking. With random 4-second evaluation windows for all models, ReDimNet2+ LMFT reduces pooled VoxCeleb1 EER from 2.42% to 0.82% and a 26-condition robustness stress-test EER from 7.21% to 1.99%. Under this shared local protocol, it reaches 0.35% EER on VoxCeleb1-O versus 0.787% for the best evaluated WeSpeaker checkpoint. On a VoxBlink2 retrieval subset, reranking improves the final model's Pr@k from 0.7413 to 0.7687.

1. INTRODUCTION

The paper asks whether a fixed compact ReDimNet2 backbone can gain robustness through changes to data, augmentation, optimization, and scoring. A VoxBlink2 subset analysis identifies differences in byte rate and predicted coloration that motivate testing codec and filtering augmentation alongside other methods.

1. INTRODUCTION

  • ASV systems can degrade under unseen channels, aggressive compression, short fragments, and retrieval-style open-set evaluation, and diverse training data alone may not cover absent degradation mechanisms.Source: §1 · p. 1
  • ReDimNet2+ keeps the ReDimNet2 architecture fixed while improving robustness through data, augmentation, optimization, and scoring; its contributions include LMFT and graph-based retrieval reranking.The introduction reports pooled VoxCeleb1 EER falling from 2.42% to 0.824%.Source: §1 · p. 1

2. DATA AND DOMAIN MISMATCH

  • The final training pool combines seven public corpora with 63,934 speakers, about 4.6 million utterances, and about 8,675 hours.The corpora contribute in-the-wild, device and dialect variation, multi-genre Chinese web speech, multilingual read speech, and Mandarin read speech.Source: §2 · p. 1
  • The VoxBlink2 analysis subset has 673,277 training files from 11,053 speakers and 134,697 evaluation files; training clips are shorter, with 22.7% under 3 seconds.Early stages use 2–3 second crops, while LMFT uses 6-second windows.Source: §2 · p. 1
  • Evaluation files show lower byte rates and more low-COL predictions than training files, motivating codec and filtering augmentation without identifying a unique physical cause.COL below 2.0 occurs in 6.2% of training files versus 41.4% of evaluation files; byte-rate variation alone does not establish channel distortion.Source: §2 · pp. 1–2

3. REDIMNET2+

ReDimNet2+ combines an efficient waveform-and-codec augmentation pipeline with staged adaptation and LMFT of the ReDimNet2 checkpoint. For retrieval, it adds a two-stage reranking approach that uses local neighbor structure and hubness penalties without retraining or changing verification scores.

3.1. Training pipeline and augmentation

  • Random-window decoding, waveform and codec augmentation, and GPU spectrogram extraction form the training pipeline, with feature augmentation applied after spectrogram computation.The codec presets model compression and telephony channels; Fig. 2 shows examples of the same utterance under different presets.Source: §3.1 · p. 2, Figure 2 · p. 3

3.2. Staged fine-tuning and LMFT

  • Adaptation starts from the public ReDimNet2 checkpoint; the final LMFT run continues from the seven-corpus adapted model and uses a fixed 0.3 margin with 6-second training windows.Other adaptation runs start independently from the public checkpoint, so the sequence characterizes the combined recipe rather than isolating individual factors.Source: §3.2 · p. 2

3.3. Retrieval reranking

  • The two reranking stages address hubness and local ranking errors in large galleries using cached embeddings and neighbor lists, without retraining or changing verification scores.Source: §3.3 · p. 2
  • Mean-chain reranking starts from a cosine-similarity candidate pool, repeatedly selects the unused candidate closest to the current probe, and updates the probe using the query and selected neighbor.Keeping the query in each update limits drift while the neighbor term follows local speaker-cluster structure.Source: §3.3 · p. 2
  • Graph reranking rescores candidates from a query's neighbors and their neighbors using rank, reciprocal support, common-neighbor support, and a hubness penalty.The final recipe uses three iterations with weights wr=1.0, wq=1.5, wc=2.0, and wh=0.3.Source: §3.3 · pp. 2–3

4. EXPERIMENTS

All systems were evaluated with random 4-second windows; ReDimNet2+ LMFT improves VoxCeleb1 verification, and reranking raises retrieval precision on the VoxBlink2 subset. The experiments also compare models under a shared local protocol and test how reranking variants affect two retrieval datasets.

4.1. Metrics and protocol

  • Verification uses EER on VoxCeleb1-O/E/H and pooled EERp, while retrieval measures Pr@k separately on VoxCeleb1 and a VoxBlink2 subset.The VoxBlink2 subset evaluation is distinct from the published open-set identification benchmark.Source: §4.1 · p. 3

4.2. Verification results

  • Under random 4-second evaluation, final LMFT reaches 0.351% EERo, 1.055% EERh, 0.824% EERp, 1.991% EERph, and 0.7413 VoxBlink2 Pr@k.EERph pools scores from 26 degraded VoxCeleb1-O variants; EERp pools VoxCeleb1-O/E/H scores.Source: §4.2 · p. 3, §4.1 · p. 3, Table 1 · p. 4

4.3. Comparison with other models

  • Under the shared 4-second protocol, ReDimNet2+ LMFT improves on the local starting checkpoint and all three evaluated WeSpeaker checkpoints on VoxCeleb1-O/E/H.Published ReDimNet2-B6 results use full utterances and are not directly comparable to the local 4-second results.Source: §4.3 · p. 3, Table 2 · p. 4

4.4. Retrieval reranking

  • On the VoxBlink2 subset, mean-chain reranking reaches 0.7220 Pr@k, graph reranking 0.7369, and their combination 0.7391.On VoxCeleb1, the combined variant gives the best Pr@45 while recovering most of the small Pr@1 and Pr@10 losses from graph-only reranking.Source: §4.4 · p. 3, Table 3 · p. 4
  • On the VoxBlink2 subset, the full reranking pipeline (mean chain + graph, same settings) has a final Pr@k of 0.7687; the graph stage improves ranking with structure invisible to pairwise scoring.Source: Table 4 · p. 4, §4.4 · p. 4

5. DISCUSSION AND CONCLUSION

ReDimNet2+ combines multi-corpus adaptation, waveform and codec augmentation, longer training windows, and LMFT; reranking improves retrieval. The development sequence does not isolate individual training-factor effects, and the reported results have important evaluation boundaries.

5. DISCUSSION AND CONCLUSION

  • ReDimNet2+ combines multi-corpus adaptation, waveform and codec augmentation, longer training windows, and LMFT, while reranking improves retrieval on VoxCeleb1 and a VoxBlink2 subset.The development sequence improves verification under the shared 4-second evaluation protocol.Source: §5 · p. 4
  • The development sequence changes several training factors, so it does not isolate their individual effects.The predicted coloration shift motivates codec and filtering augmentation but does not establish that these transformations are necessary for every target domain.Source: §5 · p. 4
  • The external-checkpoint comparison is shared-protocol rather than equal-data, EERph uses synthetic degradations, and sub-1% VoxCeleb1 EER alone does not prove in-the-wild robustness.Future work should target real telephone and VoIP data, degraded-channel calibration, and more out-of-domain benchmarks.Source: §5 · p. 4

6. COMPLIANCE WITH ETHICAL STANDARDS

The material reports study ethics and supplementary content, then reviews speaker backbones, robustness augmentation, margin-based optimization, and retrieval reranking.

6. COMPLIANCE WITH ETHICAL STANDARDS

  • The study used open-access speech corpora and publicly released pretrained speaker checkpoints, collected no new human-subject recordings, and received no funding.The authors report no relevant financial or nonfinancial interests.Source: §6 · p. 5

SUPPLEMENTARY MATERIAL (NOT PART OF THE ICASSP SUBMISSION)

9.1. Speaker Embedding Backbones

  • ReDimNet2 reshapes dimensions over time and pools speaker evidence efficiently, while this work keeps its backbone architecture fixed.Larger self-supervised encoders can have substantially higher inference cost and often require distillation or pruning for efficient deployment.Source: §9.1 · p. 6

9.2. Robustness and Augmentation

  • A predicted coloration shift in this setting motivates codec simulation and filtering alongside standard noise and reverberation augmentation.Source: §9.2 · p. 6

9.3. Margin-Based Optimization

  • Direct fine-tuning with a high fixed margin degrades clean verification, while broad adaptation followed by LMFT improves it.Other settings also change, so the development sequence does not isolate margin scheduling.Source: §9.3 · p. 6

9.4. Retrieval Reranking

  • The paper adapts reciprocal-neighbor reranking and hubness-aware similarity correction to speaker embeddings through a lightweight graph stage.Source: §9.4 · p. 6

10. TASK AND METRICS

The section defines verification, stress-test, and retrieval evaluations, then describes the VoxBlink2 subset and data characteristics relevant to training and analysis.

10.1. Verification

  • All reported verification evaluations use randomly selected 4-second windows, L2-normalized speaker embeddings, and cosine similarity; pooled EERp covers VoxCeleb1-O, -E, and -H.EER is the threshold operating point where FAR and FRR are equal or closest under the discrete score set.Source: §10.1 · p. 6

10.2. Robustness Stress Test

  • EERph pools scores from 26 degraded variants of VoxCeleb1-O utterances into one stress-test EER.The metric tests speaker separation across many channel transformations simultaneously.Source: §10.2 · p. 6

10.3. Speaker Retrieval

  • Retrieval uses Precision@k on VoxCeleb1 and a VoxBlink2 subset; the latter is distinct from the published open-set identification benchmark.VoxCeleb1 Pr@k values are percentages, whereas VoxBlink2-subset values are fractions; verification and retrieval diagnose different behavior.Source: §10.3 · p. 6

11.1. Training Corpora

  • The final training pool combines seven corpora, including in-the-wild speech, device and dialect variation, multilingual read speech, and Mandarin subdialect recordings.Checkpoints are selected on VoxCeleb1 development data, which is also used for verification and retrieval evaluation.Source: §11.1 · pp. 6–7

11.2. VoxBlink2 Subset Overview

  • The VoxBlink2 subset contains 673,277 training files from 11,053 speakers and 134,697 evaluation files; speaker counts are imbalanced.The training and evaluation audio is 16 kHz mono; the training split includes both very lightly and heavily represented speakers.Source: §11.2 · p. 7

11.3. Duration Mismatch

  • Training utterances are shorter on average, so early training uses 2–3-second windows and LMFT uses 6-second windows with repeat padding for short files.The training split has 22.7% of utterances below 3 seconds, while the evaluation subset contains more 5–20-second clips.Source: §11.3 · p. 7

11.4. Coloration and Byte-Rate Shift

  • Median FLAC byte rate drops from 21,040 bytes/s in train to 14,021 bytes/s in the evaluation subset, while COL below 2.0 rises from 6.2% to 41.4%.Byte rate is a heuristic rather than a direct bandwidth or codec-degradation measurement; the coloration shift motivates testing filtering and codec simulation without establishing a unique physical cause.Source: §11.4 · p. 7

11.5. Within-Speaker Channel Variation

  • For 884 speakers with at least five recordings, median within-speaker byte-rate standard deviation is 1,753, consistent with heterogeneous recording conditions.Byte-rate variation alone cannot establish channel variation or positive-pair difficulty; the observation motivates compact representations across recording conditions.Source: §11.5 · p. 7

12. TRAINING PIPELINE

The training pipeline uses manifest-based random-window decoding, preprocessing speed improvements, and waveform, codec, and feature-level augmentation.

12.1. Manifest and Random-Window Decoding

  • The loader indexes manifest metadata, selects a fixed-length window for decoding, and falls back to a neighboring valid example when decoding fails.Audio is converted to mono 16 kHz, and short segments are repeat-padded to avoid long artificial silence tails during LMFT.Source: §12.1 · p. 7

12.2. Pipeline Speed

  • Range decoding is 2.6× faster than decoding full files and cropping afterward.Other speed measures include fixed speed-perturbation factors, direct FFmpeg calls for codec augmentation, and GPU spectrogram computation.Source: §12.2 · p. 7

12.3. Waveform and Codec Augmentation

  • Examples are assigned among clean, codec-only, waveform-only, or combined augmentation modes, with probabilities 0.5, 0.2, 0.2, and 0.1.Waveform transformations include filtering, noise, room responses, and speed perturbation; codec presets simulate compression and telephony channels, while SpecAugment and CutMix are applied at feature level.Source: §12.3 · p. 7

13. MODEL AND OPTIMIZATION

ReDimNet2+ adapts a fixed compact ReDimNet2 backbone through multi-corpus fine-tuning and augmentation, then applies large-margin fine-tuning. The section describes the classification objective and training-window settings used in these stages.

13.1. Backbone

  • ReDimNet2 alternates local aggregation and dimensional reshaping to capture time-frequency structure with moderate compute, starting from a pretrained checkpoint.Fig. 3 illustrates waveform transformations including colored noise, RawBoost, band-pass filtering, and speed perturbation.Source: §13.1 · p. 7, Figure 3 · p. 8

13.2. Fine-Tuning Stages

  • Adaptation runs start independently from the public checkpoint except final LMFT, which continues from the seven-corpus model with a fixed 0.3 margin and 6-second windows.Earlier margin comparisons do not isolate margin scheduling because learning rates and training duration also differ.Source: §13.2 · p. 8

13.3. Speaker Classification Objective

  • SphereFace2 trains one-versus-all binary classifiers for normalized embeddings and speaker-class weights; the final LMFT stage uses a fixed 0.3 margin and 6-second segments.All models use randomly selected 4-second windows for evaluation.Source: §13.3 · p. 8

13.4. Hyperparameters

  • Training windows increase from 32,200 samples in early runs to 48,300 in later runs, while LMFT uses 96,000-sample windows.Training used distributed execution with Accelerate and FSDP, BFloat16 precision, AdamW, gradient clipping at 20, and fixed seed 42.Source: §13.4 · p. 9

14. EXTERNAL BENCHMARK PROTOCOL

The benchmark compares public embedding checkpoints using a shared local VoxCeleb1 protocol with randomly selected 4-second windows. Retrieval reranking first builds a mean-chain neighbor list, then rescores candidates using local graph structure.

14.1. Model Selection

  • The external comparison uses public WeSpeaker ResNet34-LM, ECAPA512-LM, and CAM++ ONNX checkpoints selected for practical local inference.The models represent different open-source speaker-embedding families.Source: §14.1 · p. 9

14.2. Local VoxCeleb1 Setup

  • All local systems use the same VoxCeleb1 trial files and randomly selected 4-second windows, with checkpoints selected on the VoxCeleb1 development set.The published ReDimNet2-B6 reference instead uses full utterances and cosine scoring without score normalization, so the local numbers are protocol-specific.Source: §14.2 · p. 9

15.1. Mean-Chain Reranking

  • Mean-chain reranking selects candidates within an initial cosine-similarity pool, updating each probe as the normalized mean of the query and latest selected neighbor.The update is not cumulative; anchoring each probe to the query limits drift while allowing the order to follow local embedding structure.Source: §15.1 · p. 9

15.2. Graph Reranking

  • Graph reranking uses a label-free K-nearest-neighbor graph, scoring current neighbors and their neighbors by rank, reciprocal support, common neighbors, and hubness penalty.The final experiments use three iterations with unchanged weights, rebuilding the graph after each rescoring stage.Source: §15.2 · p. 9

16. RESULTS

ReDimNet2+ LMFT improves verification across the reported VoxCeleb1 protocols, while the study also examines training-data scaling, external checkpoints, retrieval reranking, and inference runtime. Its published-baseline comparison is described as indicative rather than equal-budget.

16.1. Main Verification Results

  • 0.824% pooled EER (from 2.42%) and 1.991% EERph: ReDimNet2+ LMFT also reaches 0.351% EERo and 1.055% EERh under shared 4-second evaluation.The final run reaches 0.351% EERo, 1.055% EERh, 0.824% EERp, and 1.991% EERph.Source: §16.1 · p. 9

16.2. Published VoxCeleb1 Baselines

  • ReDimNet2+ is competitive with strong VoxCeleb2-dev systems but trails the extended-data W2V-BERT 2.0 result [22]; published comparisons are indicative, not equal-budget.The paper identifies its reproducible local comparison as the controlled contribution rather than claiming universal state of the art.Source: §16.2 · p. 10

16.3. Data Scaling

  • VoxCeleb1-O shows the most stable monotonic EER trend as training speakers and utterances increase, while VoxCeleb1-E and -H become less monotonic at the largest settings.The paper notes that raw scale helps, but domain and quality of added data also influence extended and hard protocols.Source: §16.3 · p. 10

16.4. External Baselines

  • 0.351 / 0.523 / 1.055% EER on VoxCeleb1-O/E/H is lower than all three reported WeSpeaker ONNX checkpoints under the shared local 4-second protocol.Training budgets and corpora differ, so this is a local-system comparison rather than a matched-training-data comparison.Source: §16.4 · p. 10

16.5. Reranking Results

  • 0.7687 Pr@k on the VoxBlink2 subset is reached by reranking the final LMFT system, compared with 0.6906 for baseline ReDimNet2 after the same reranking.On the pretrained setup, mean-chain, graph, and combined reranking reach 0.7220, 0.7369, and 0.7391 Pr@k, respectively, from a 0.7024 baseline.Source: §16.5 · p. 10

16.6. Inference

  • 230.66× real time is reached by TensorRT FP16 on fixed 6-second segments, with EER close to Torch FP32 in the same setup.These accuracy figures are separate from the main 4-second evaluation.Source: §16.6 · p. 10

17. DISCUSSION

The discussion attributes improved verification and retrieval to a combined training recipe, while treating codec augmentation as motivated by observed shifts rather than universally necessary. It also emphasizes remaining robustness gaps and limits on what the evaluations establish.

17.1. Main Sources of Improvement

  • Broader training data, longer training windows, and LMFT accompany verification and retrieval improvements, but the development sequence does not isolate each intervention’s causal effect.The evaluation window remains 4 seconds; LMFT uses 6-second training segments with a stronger margin.Source: §17.1 · p. 10

17.2. Motivation for Codec Augmentation

  • The observed coloration and FLAC-compressibility shifts motivate codec simulation and filtering alongside noise and reverberation, but do not establish codec augmentation as necessary for every target domain.FLAC encoding is lossless and should not be interpreted as lossy-codec corruption.Source: §17.2 · p. 10

17.3. Verification Versus Retrieval

  • Graph reranking improves retrieval by using reciprocal neighbors, shared neighbors, and hubness penalties—structure not captured by pairwise scoring alone.EER measures threshold behavior on labeled pairs, whereas Pr@k measures ranking quality in a gallery.Source: §17.3 · p. 10

17.4. Remaining Failure Modes

  • The hard-impostor protocol and synthetic codec-and-waveform stress suite still expose speaker confusions and degraded-channel errors after LMFT.The discussion points to real telephone/VoIP data, codec curriculum design, and calibration under degraded-channel score distributions as future work.Source: §17.4 · p. 10
  • Published VoxCeleb1 results use full utterances, while the local comparison uses 4-second windows, so the protocols are not directly comparable.The separate inference-speed and quality benchmark uses fixed 6-second segments; xRTF is audio duration divided by processing time, with higher values indicating faster processing.Source: Table 11 · p. 11, Table 12 · p. 11, Table 13 · p. 11

17.5. Limitations

  • The local comparison with external checkpoints is not an equal-training-data architecture comparison, and synthetic degradations need not cover all real-world channel failures.The discussion also identifies difficult open-set out-of-domain generalization as a remaining challenge; sub-1% VoxCeleb1 EER alone is insufficient evidence of robustness for arbitrary in-the-wild deployment.Source: §17.5 · pp. 11–12

18. REPRODUCIBILITY

The reproducibility materials include public code and weights, recorded training artifacts, and a documented local benchmark with checks and environment details.

18. REPRODUCIBILITY

  • Code and model weights are publicly listed at the project’s GitHub and Hugging Face locations.The passage provides github.com/lab260ru/redimnet2-plus and huggingface.co/lab260/redimnet2-plus.Source: §18 · p. 12
  • Recorded artifacts include model code, per-stage configurations, and run-level hyperparameters, alongside fixed-seed, mixed-precision, optimizer, clipping, and distributed-execution settings.The experiments use seed 42, BFloat16, AdamW, gradient clipping at 20, Accelerate, and FSDP.Source: §18 · p. 12
  • The external benchmark records checksums, scores, embeddings, aggregate tables, and environment details, and its code is covered by nine unit tests.The tests cover trial parsing, EER computation, audio loading, shard partitioning, O/E/H aggregation, and embedding-cache reuse, among other checks.Source: §18 · p. 12
Loading 2609.37014v1…