PaperlayerView exact revision on arXiv

Source-linked AI summary

LASE: Language-Adversarial Speaker Encoding for Indic Cross-Script Identity Preservation

Venkata Pushpak Teja Menta

arXiv:2605.00777v1cs.SDcs.CLeess.AS

TL;DR

Speaker encoders may represent the same voice differently across Indic scripts, challenging a load-bearing assumption in cross-script voice cloning. LASE applies language-adversarial training to a projection head over frozen WavLM-base-plus, and its residual cross-script gap is consistent with zero on both evaluated corpora.

  • Problem

    The paper asks whether speaker-verification encoders preserve identity across Indic scripts, a load-bearing assumption for cross-script voice cloning and Indian conversational-audio diarisation.

    Source: §I · p. 1

  • Method

    LASE applies gradient-reversal training against a language classifier to a small projection head over frozen WavLM-base-plus.

    Source: §I · p. 1

  • Results

    ∆ = 0.013 [−.02,.05] (Western) and 0.026 [−.04,.08] (Indian) for WavLM+GRL, with both confidence intervals including zero.

    Source: E. Ablation: GRL contribution vs… · p. 5

  • Takeaways & Limitations

    The results support explicit invariance as a sample-efficient route to Indic cross-script consistency compared with relying on corpus scale alone.

    Source: §VII · p. 6

    Training and held-out audio are synthetic and both come from ElevenLabs Multilingual, so the evaluated gap does not establish performance on real-world cross-script speech.

    Source: §VII · p. 6

Abstract

from arXiv · show

A speaker encoder used in multilingual voice cloning should treat the same speaker identically regardless of which script the audio was uttered in. Off-the-shelf encoders do not, and the failure is accent-conditional. On a 1043-pair Western-accented voice corpus across English, Hindi, Telugu, and Tamil, WavLM-base-plus-sv loses 0.082 absolute cosine similarity when the same voice changes script and ECAPA-TDNN loses 0.105. On a 1369-pair Indian-accented voice corpus, the gap shrinks to 0.006 (WavLM-SV) and 0.044 (ECAPA-TDNN). The leak is largest where it matters most for cross-script TTS: when a system projects a non-Indic-trained voice into Indic scripts. We present LASE (Language-Adversarial Speaker Encoder), a small projection head over frozen WavLM-base-plus trained with two losses: a supervised contrastive loss over voice identity, and a gradient-reversal cross-entropy against a 4-language classifier that pushes the embedding to be language-uninformative while remaining speaker-informative. Trained on 1118 quality-gated cross-script pairs synthesised from 8 commercial multilingual voices, LASE's residual gap is consistent with zero on both corpora (Delta = 0.013 Western, Delta = 0.026 Indian; both bootstrap 95% CIs include zero) and amplifies the cross-script-vs-floor margin 2.4-2.7x over both baselines. An ECAPA+GRL ablation shows the GRL objective improves either backbone but the WavLM choice contributes too. In synthetic multi-speaker diarisation, LASE matches ECAPA-TDNN on cross-script speaker recall (0.788 vs 0.789) with ~100x less training data. We release the r1 checkpoint, both corpora, and the bootstrap recipe.

I. Introduction · A. Contributions · II. Related Work

LASE targets script-dependent identity drift that undermines Indic cross-script cloning and diarisation. On held-out synthetic evaluation, it reduces the WavLM cross-script drop from 0.082 to 0.013, though transfer to natural speech and new voices remains untested.

I. Introduction

  • Cross-script identity consistency is a load-bearing assumption for Indic voice cloning and diarisation, yet existing encoders fail noticeably on Indic scripts.The failure can split one speaker across Hindi and English in a call or make cross-script cloning sound like a different voice doing an accent.Source: §I · p. 1
  • WavLM-base-plus-sv median same-voice similarity drops from 0.927 same-script to 0.845 cross-script, while ECAPA-TDNN drops from 0.499 to 0.394.The respective absolute drops are 0.082 and 0.105 on the same test.Source: §I · p. 1
  • LASE applies gradient-reversal training against a language classifier to a projection head over frozen WavLM-base-plus; its evaluation uses synthetic, content-held-out audio from voices also seen in training.The authors report no evidence yet of closure transferring to natural human speech or of generalisation to new voices.Source: §I · p. 1

A. Contributions

  • 0.082 absolute drop in WavLM cross-script similarity falls to 0.013 with LASE, closing 84.3% of the gap on held-out evaluation.LASE r1 uses a 256-dimensional embedding built from frozen WavLM-base-plus and a projection head with a gradient-reversal language classifier.Source: A. Contributions · p. 2
  • LASE matches ECAPA-TDNN on synthetic cross-script diarisation speaker recall (0.788 vs 0.789) despite roughly 100× less training data.LASE trains on 1118 pairs, compared with ECAPA’s 1M+ VoxCeleb utterances.Source: A. Contributions · p. 2
  • The released benchmark contains 1118 training and 1043 held-out pairs across 8 voices and 4 languages, with within-script, cross-script, and across-speaker distributions.Released artefacts include LASE r1 weights under MIT, corpora under CC-BY-4.0, and scoring scripts.Source: A. Contributions · p. 2

II. Related Work

  • Prior work applies adversarial methods to channel-invariant or English-European speaker representations, while LASE adapts the established recipe to the Indic cross-script setting.The paper presents the technique as borrowed, with the Indic application, corpus, and empirical demonstration as new.Source: §II · p. 2
  • Because large natural same-speaker cross-script corpora are unavailable, the study synthesises matched-content speech and quality-gates pairs, dropping 30% when identity appears lost.The pass rates are 1118/1600 training pairs and 1043/1600 held-out pairs; PSR measures English/British accent shifts, whereas Pairwise Accent Similarity measures accent rather than identity and PSP addresses phonological accent dimensions.Source: §II · p. 2

III. Method · A. Architecture · B. Losses

LASE combines frozen WavLM-base-plus features with a trainable speaker projection and a gradient-reversal language classifier. Its losses promote voice-identity discrimination while making embeddings uninformative about language.

A. Architecture

  • LASE produces 256-dimensional speaker embeddings using a frozen WavLM-base-plus backbone, a trainable projection head, and a gradient-reversal language classifier.Source: A. Architecture · p. 2
  • At 16 kHz, frozen WavLM-base-plus is truncated at the transformer stack to produce T × 768 frame-level features.Source: A. Architecture · p. 2
  • The projection head mean-pools layers 10–12, then applies a 768 →512 →256 two-layer MLP with ReLU and dropout 0.1.Source: A. Architecture · p. 2
  • A scheduled gradient-reversal layer scales classifier gradients by −λt, training the projection head to hide which of {en, hi, te, ta} each clip belongs to.Source: A. Architecture · p. 2

B. Losses

  • SupCon pulls together same-voice items and separates different voices regardless of script; mixed-script batches make voice identity its only usable signal, with τ = 0.07.Source: B. Losses · pp. 2–3
  • The language adversary uses 4-class cross-entropy, and the total objective is L = Lspk + λtLlang with λt ramped according to a schedule.Source: B. Losses · p. 3
  • With gradient reversal, Llang near ln 4 ≈1.386 indicates language is hidden, whereas Llang near 0 indicates language identity leaks perfectly.Source: B. Losses · p. 3

C. λ schedule and training setup

Training uses a gradual GRL-strength schedule so SupCon can first shape voice-coherent embeddings, alongside a specified optimization and reproducibility setup.

C. λ schedule and training setup

  • λ is 0 for 200 warmup steps, ramps linearly to 0.1 over 500 steps, then stays at 0.1 to avoid disrupting early SupCon geometry.The schedule is intended to let SupCon establish voice-coherent structure before adversarial training begins.Source: C. λ schedule and training setup · p. 3
  • AdamW training uses learning rate 10^-4, weight decay 0.01, batch size 16, gradient clipping at 1.0, and 1000 steps on one A10G GPU for $0.31.Batches mix voices and languages to include same-voice positives and across-voice negatives; the run takes about 17 minutes at $1.10/hr.Source: C. λ schedule and training setup · p. 3
  • The verified software chain pins torch and torchaudio 2.4.0, transformers 4.49.0, peft 0.13.0, accelerate 0.30.0, CUDA 12.1, and seed 1337.Seed 1337 is used for training, evaluation pair sampling, and held-out source generation.Source: C. λ schedule and training setup · p. 3

IV. Corpus construction … C. Train / held-out split discipline

Because natural Indic corpora lack large-scale same-speaker recordings across scripts, the paper bootstraps a gated synthetic corpus and uses a held-out split to test generalization to new utterances from the same voices.

A. The data problem

  • Available natural Indic corpora do not provide same-speaker recordings across scripts at scale; FLEURS explicitly splits speakers by language.IndicTTS speakers are mostly monolingual, Common Voice 17 has no enforced cross-language overlap, and collecting a native multilingual held-out set has a cost.Source: A. The data problem · p. 3
  • The corpus uses eight verified-multilingual ElevenLabs voices and restricts synthesis to English, Hindi, Telugu, and Tamil.English provides a Latin-script reference; Hindi is the most-resourced Indic language, while Telugu and Tamil are large Dravidian languages where English-trained SV encoders are known to drift hardest [11].Source: A. The data problem · p. 3
  • For each voice, 50 sentences per language yield 1600 nominal clips, filtered by a WavLM-base-plus-sv cosine threshold of 0.90 against the voice’s first English sample.Bengali, Kannada, Gujarati, and Malayalam are deferred to v2 to keep the corpus small enough for free-tier ElevenLabs credits.Source: A. The data problem · p. 3

C. Train / held-out split discipline

  • The cosine gate retained 1118/1600 training clips (70%) and 1043/1600 held-out clips (65%), using different sentences but the same eight voices.The paper reports deterministic held-out data reproducible from paper/lase/source_heldout.jsonl and uses the same audio distribution for apples-to-apples encoder comparisons.Source: A. The data problem · p. 3, C. Train / held-out split discipline · p. 3

B. Why the 30% drop is informative, not a bug

  • 30% of synthesis attempts failed the cosine gate, and visible inspection showed identity drift across scripts in these clips.The authors treat this failure rate as an audit signal and a concrete floor for cross-script identity degradation in current systems.Source: B. Why the 30% drop is informative,… · p. 3

C. Train / held-out split discipline

  • Training uses 1118 pairs and held-out evaluation uses 1043 pairs from the same eight voices, with new sentences absent from training.This tests content generalization, not generalization to entirely new voices, which is deferred to v2.Source: C. Train / held-out split discipline · p. 3

V. Three-distribution measurement framework

The framework separates same-speaker similarity across scripts from within-script similarity and a different-speaker noise floor. It summarizes script-related identity loss with a gap and retained speaker separability with a margin.

V. Three-distribution measurement framework

  • Three cosine-similarity distributions compare same-voice clips within one script, same-voice clips across scripts, and different-voice clips within one language.For each distribution, 200 random qualifying pairs are sampled and the median cosine is reported.Source: §V · p. 3
  • The gap ∆ is medianwithin − mediancross, while margin M is mediancross − medianacross; lower gap and higher margin are better.Within-script similarity is the upper bound, across-speaker similarity the lower bound, and cross-script similarity should fall between them.Source: §V · p. 4
  • A good cross-script encoder keeps ∆ small while making M large, preserving identity similarity across scripts and separation from different speakers.Source: §V · p. 4

VI. Results · A. Headline: three-encoder comparison on two held-out corpora · the off-the-shelf WavLM-SV (Western) and ECAPA-TDNN (both

LASE sharply narrows the Western-corpus within-versus-cross-script gap and widens the cross-script-versus-floor margin compared with both baselines. Its gap confidence interval includes zero on both held-out corpora.

A. Headline: three-encoder comparison on two held-out corpora

the off-the-shelf WavLM-SV (Western) and ECAPA-TDNN (both

A. Headline: three-encoder comparison on two held-out corpora

the off-the-shelf WavLM-SV (Western) and ECAPA-TDNN (both

  • For the Western corpus, Fig. 1 shows LASE’s within- and cross-script distributions nearly touching while its across-speaker floor sits well below cross-script scores.Source: Figure 1 · p. 4

B. Training dynamics confirm the loss design

Across 1,000 training steps, LASE’s loss curves show speaker-coherent embeddings emerging while the language classifier remains at chance; the λt schedule shapes this disentanglement.

B. Training dynamics confirm the loss design

  • Lspk drops from ∼2.7 to ∼0.5–1.0 by step 900, while Llang stays near ln 4 ≈1.386 throughout training.The losses indicate that the projection head learns voice-coherent geometry while the language classifier never beats random, leaving no exploitable language information in z.Source: B. Training dynamics confirm the loss… · p. 4
  • λt ramps from 0 to 0.1 over the first 700 steps and holds; earlier ramps caused SupCon collapse, while later ramps slowed language disentanglement.The paper describes this schedule as empirically the cleanest of the three tried.Source: B. Training dynamics confirm the loss… · p. 4

C. Diarisation: matching ECAPA on cross-script recall with 100× less data · D. Generalisation: held-out vs training corpus

LASE nearly matches ECAPA-TDNN on cross-script diarisation recall with far less training data. Its cross-script gap closure is larger on held-out data than on training data, indicating generalisation to fresh content from familiar voices.

C. Diarisation: matching ECAPA on cross-script recall with 100× less data

  • The synthetic benchmark contains 50 conversations totaling 23.7 minutes, with 411 segments built from 2–4 voices and 6–10 held-out clips per conversation.A voice may switch languages mid-conversation; ground-truth speaker labels are provided in RTTM format.Source: C. Diarisation: matching ECAPA on… · p. 4
  • Each segment is clustered using cosine-distance agglomerative clustering with K set to the true speaker count, and evaluated by ARI and cross-script recall.No VAD model is included: constructed segment boundaries are treated as known to isolate encoder clustering quality.Source: C. Diarisation: matching ECAPA on… · p. 5
  • 0.788 vs 0.789 cross-script speaker recall statistically ties LASE with ECAPA-TDNN, although ECAPA leads overall ARI, 0.693 vs 0.640.The comparison is on a synthetic 50-conversation code-switching benchmark.Source: Table II · p. 5, C. Diarisation: matching ECAPA on… · p. 5
  • LASE matches ECAPA on cross-script recall using 1118 same-voice cross-script pairs, versus ECAPA’s roughly 1M VoxCeleb utterances—over 100× more training data.The passage attributes ECAPA’s scale to broader speaker, microphone, recording-condition, and language coverage.Source: C. Diarisation: matching ECAPA on… · p. 5
  • 0.604 cross-script recall and 0.444 ARI for WavLM-base-plus-sv leave both LASE and ECAPA materially ahead.The passage describes both methods as crushing this off-the-shelf baseline.Source: C. Diarisation: matching ECAPA on… · p. 5

D. Generalisation: held-out vs training corpus

  • 84.3% relative held-out gap closure (∆ from 0.082 to 0.013) exceeds the 78.7% training-set closure (∆ from 0.099 to 0.021).Both are scored against their own WavLM-SV baselines; the result supports generalisation to fresh content from familiar voices rather than memorisation of training pairs.Source: D. Generalisation: held-out vs… · p. 5

E. Ablation: GRL contribution vs backbone contribution · F. The cross-script gap is accent-conditional

LASE brings the cross-script gap close to zero in both accent regimes, while the off-the-shelf gap is much larger for Western-accented voices. The ablation indicates that GRL helps both backbones, while WavLM and GRL together deliver the full effect across both corpora.

E. Ablation: GRL contribution vs backbone contribution

  • The ECAPA+GRL ablation used the same 1000-step recipe, losses, λt schedule, hyperparameters, and 1118-pair corpus as LASE, swapping only the frozen WavLM backbone for frozen SpeechBrain ECAPA-TDNN.The comparison was added as a fourth row in Table I.Source: E. Ablation: GRL contribution vs… · p. 5
  • GRL reduces ECAPA-TDNN’s gap from 0.105 to 0.027 (75% relative) on Western voices and from 0.044 to 0.037 (16% relative) on Indian voices.The effect transfers across backbones and is larger on the Western corpus.Source: E. Ablation: GRL contribution vs… · p. 5
  • ∆ is 0.013 [−.02,.05] vs 0.027 [−.02,.08] Western and 0.026 [−.04,.08] vs 0.037 [−.03,.10] Indian for WavLM+GRL vs ECAPA+GRL; CIs overlap.The corresponding M margins are 0.662 vs 0.739 Western and 0.344 vs 0.247 Indian, favoring WavLM+GRL on Indian voices.Source: E. Ablation: GRL contribution vs… · p. 5
  • Neither GRL-on-ECAPA (best gap 0.027) nor WavLM without GRL (Western gap 0.082; Indian gap 0.006) matches LASE across both corpora.The Indian WavLM-SV-only gap of 0.006 is an accent-conditional ceiling effect; the combination of WavLM and GRL recovers the full LASE effect across both corpora.Source: E. Ablation: GRL contribution vs… · p. 5
  • Under GRL training, WavLM’s language loss stayed at ln 4 ≈1.386, while ECAPA’s oscillated between 0.4 and 2.5; eval-time gaps were 0.013 vs 0.027.This indicates that language invariance was harder to achieve with ECAPA under the same adversarial pressure.Source: E. Ablation: GRL contribution vs… · pp. 5–6

F. The cross-script gap is accent-conditional

  • ∆ = 0.006 for WavLM-SV on Indian-accented voices versus ∆ = 0.082 on Western-accented voices, a 14× contrast linked to shared Indian phonetic structure.The practical consequence is greatest when a non-Indic-trained input voice is projected into Indic scripts.Source: F. The cross-script gap is… · p. 6
  • 70% of Western-voice cross-script pairs passed the WavLM-SV ≥0.90 quality gate, versus 86% of Indian-voice pairs.The passage attributes the difference to better identity preservation for Indian-accented voices in ElevenLabs Multilingual.Source: F. The cross-script gap is… · p. 6
  • Δ = 0.013 Western and Δ = 0.026 Indian, with both bootstrap CIs straddling zero; LASE’s margin advantages are 2.7×/3.3× over WavLM-SV/ECAPA-TDNN Western and 2.4×/1.3× Indian.LASE preserves discrimination where WavLM-SV’s margin drops from 0.245 to 0.141 between Western and Indian corpora.Source: F. The cross-script gap is… · p. 6

VII. Discussion · VIII. Reproduction

LASE’s cross-script recall result supports the value of explicit language invariance, but its evidence is limited by synthetic data and shared training/held-out voices. The authors release weights, corpora, code, and scripts for reproduction.

VII. Discussion

  • LASE matches a 100×-larger ECAPA-TDNN encoder on cross-script speaker recall, consistent with explicit invariance being more sample-efficient than relying on corpus scale.The authors attribute the strong invariance signal to the distance among the four scripts and the tight cross-script same-speaker constraint.Source: §VII · p. 6
  • VoxCeleb’s English-heavy training distribution shapes WavLM and ECAPA representations around English phones, while LASE’s GRL discourages embeddings from depending on a language’s phone inventory.The passage cites Telugu retroflex stops and aspirated consonants as phones outside the English-shaped representation.Source: §VII · p. 6
  • LASE’s evidence is limited by synthetic-only training and held-out audio, held-out sentences sharing voices with training, and lower overall ARI than ECAPA.The authors leave real-world variability and new-voice generalisation for future work, and state LASE is aimed at cross-script consistency rather than general speaker verification.Source: §VII · p. 6
  • The authors did not test IndicWhisper-style or Sarvam encoders, MOS or listening ratings, or codemixed clips; they identify cross-script cloning, diarisation, and analytics as deployment targets.They state that LASE’s 84% gap closure and matched-ECAPA cross-script recall could reduce hallucinated speaker changes at script transitions and tighten cross-script clusters.Source: §VII · p. 6

VIII. Reproduction

  • The public MIT repository provides LASE r1 weights and the 1118-pair training corpus, with the corpus released under CC-BY-4.0.The repository is at https://github.com/praxelhq/lase; weights are on Hugging Face as Praxel/lase-r1, and the corpus as Praxel/codeswitch-pairs-lase.Source: §VIII · pp. 6–7
  • The release includes Western held-out and Indian-accent held-out corpora of 1043 and 1369 pairs, respectively, plus training and bootstrap-CI scripts.The passage identifies scripts for training on a Modal A10G and for generating the confidence intervals reported in Table I.Source: §VIII · p. 7
  • The release also includes ECAPA+GRL and diarisation evaluation scripts, and a complete v1 reproduction runs on one A10G in about 25 minutes.The listed tools include the ablation evaluator, diarisation benchmark builder, and diarisation scorer.Source: §VIII · p. 7

IX. Conclusion

LASE r1 reduces language-related speaker-identity leakage across scripts while matching ECAPA-TDNN on cross-script speaker recall using far less training data.

IX. Conclusion

  • LASE r1 closes the within-vs-cross-script gap by 84% relative, doubles the speaker-discrimination margin, and matches ECAPA-TDNN on cross-script speaker recall with 100× less training data.The comparison used held-out audio and synthetic multi-speaker conversations.Source: §IX · p. 7
  • LASE r1 uses a 256-dim projection over frozen WavLM-base-plus, trained with supervised contrastive and gradient-reversal language-classifier objectives on 1118 pairs from 8 ElevenLabs Multilingual voices.The pairs were same-voice cross-script pairs synthesised for training.Source: §IX · p. 7
  • Off-the-shelf speaker encoders lose 8–11% of absolute cosine similarity when the same speaker changes Indic script, motivating a small, reproducible recipe for cross-script invariance.The recipe applies to script sets where TTS-bootstrapped pairs can be cleanly synthesised.Source: §IX · p. 7

Related papers

Loading 2605.00777v1…