PaperlayerView exact revision on arXiv

Source-linked AI summary

LASE: Language-Adversarial Speaker Encoding for Indic Cross-Script Identity Preservation

Venkata Pushpak Teja Menta

arXiv:2605.00777v1cs.SDcs.CLeess.AS

TL;DR

Off-the-shelf speaker encoders entangle language with identity across Indic scripts, weakening same-speaker recognition when scripts change. LASE addresses this with contrastive speaker learning and gradient-reversal language adversarial training over frozen WavLM, closing the gap on synthetic evaluation while remaining untested for natural speech and new voices.

  • Problem

    Off-the-shelf speaker encoders lose speaker-identity consistency across Indic scripts, a gap that matters for cross-script voice cloning and diarisation.

  • Method

    LASE trains a projection head over frozen WavLM-base-plus with supervised contrastive voice-identity loss and gradient-reversal cross-entropy against a four-language classifier.

  • Results

    0.013 Western and 0.026 Indian residual gaps are consistent with zero, while LASE matches ECAPA-TDNN on cross-script speaker recall at 0.788 vs 0.789.

  • Takeaways & Limitations

    LASE provides cross-script speaker consistency comparable to ECAPA-TDNN in synthetic diarisation with roughly 100× less training data.

  • Takeaways & Limitations

    Training and held-out evaluation use synthetic ElevenLabs audio and share the same eight voices, so natural-speech and new-voice generalisation remain untested.

Abstract

from arXiv · show

A speaker encoder used in multilingual voice cloning should treat the same speaker identically regardless of which script the audio was uttered in. Off-the-shelf encoders do not, and the failure is accent-conditional. On a 1043-pair Western-accented voice corpus across English, Hindi, Telugu, and Tamil, WavLM-base-plus-sv loses 0.082 absolute cosine similarity when the same voice changes script and ECAPA-TDNN loses 0.105. On a 1369-pair Indian-accented voice corpus, the gap shrinks to 0.006 (WavLM-SV) and 0.044 (ECAPA-TDNN). The leak is largest where it matters most for cross-script TTS: when a system projects a non-Indic-trained voice into Indic scripts. We present LASE (Language-Adversarial Speaker Encoder), a small projection head over frozen WavLM-base-plus trained with two losses: a supervised contrastive loss over voice identity, and a gradient-reversal cross-entropy against a 4-language classifier that pushes the embedding to be language-uninformative while remaining speaker-informative. Trained on 1118 quality-gated cross-script pairs synthesised from 8 commercial multilingual voices, LASE's residual gap is consistent with zero on both corpora (Delta = 0.013 Western, Delta = 0.026 Indian; both bootstrap 95% CIs include zero) and amplifies the cross-script-vs-floor margin 2.4-2.7x over both baselines. An ECAPA+GRL ablation shows the GRL objective improves either backbone but the WavLM choice contributes too. In synthetic multi-speaker diarisation, LASE matches ECAPA-TDNN on cross-script speaker recall (0.788 vs 0.789) with ~100x less training data. We release the r1 checkpoint, both corpora, and the bootstrap recipe.

ORCID: 0009-0003-2479-9208

LASE targets the failure of speaker encoders to preserve identity across Indic and Latin scripts, using adversarial language-invariant projection over WavLM. It closes the measured cross-script gap on synthetic held-out evaluation, while remaining scoped to synthetic data and shared voices.

  • Indic cross-script identity preservation fails noticeably in off-the-shelf speaker encoders, affecting voice cloning and diarisation.
  • 0.082 absolute cosine similarity is lost by WavLM-base-plus-sv between same-voice same-script and different-script clips on the 1043-pair corpus.
  • LASE combines a frozen WavLM-base-plus backbone, a projection head, supervised contrastive learning, and gradient-reversal language classification.
  • The evaluation is limited to synthetic ElevenLabs audio and held-out sentences from voices also appearing during training.
  • 84.3% relative reduction closes the held-out WavLM cross-script identity gap, reducing the 0.082 absolute drop to 0.013.

II. Related Work

The paper adapts established adversarial speaker-representation methods to an Indic cross-script setting and builds a targeted TTS-bootstrapped benchmark and LASE encoder.

  • Related work: ECAPA-TDNN and WavLM-base-plus-sv are English-pretrained speaker encoders that inherit language–identity entanglement across Indic scripts.
  • Related work: Gradient-reversal training is an established domain-adversarial recipe, previously applied to channel invariance and multilingual speaker recognition.
  • Related work: The paper distinguishes identity preservation from accent benchmarks and proposes that LASE and phonological accent evaluation can compose.
  • Benchmark construction: The corpus is TTS-bootstrapped from eight multilingual voices because no natural same-speaker cross-script corpus exists at scale.
  • LASE architecture: LASE uses a frozen WavLM-base-plus backbone, a 768→512→256 projection head, and a four-language gradient-reversal classifier.

B. Losses

LASE optimizes speaker identity and language invariance jointly: supervised contrastive learning organizes voice identities, while gradient reversal suppresses language information.

  • Speaker loss: The supervised contrastive loss pulls together clips sharing voice identity and separates clips from different voices regardless of script.
  • Speaker loss: The positive set P(i) contains same-voice items, with temperature τ = 0.07.
  • Language loss: The language adversarial loss applies four-class cross-entropy to the language classifier output.
  • Joint objective: L = Lspk + λtLlang combines the two losses, with language loss near ln 4 ≈1.386 indicating that language is hidden from the classifier.
  • Optimization: λt is warmed up at zero, ramped to 0.1, and then held constant to avoid destabilizing speaker geometry.
  • Optimization: Training uses AdamW for 1000 steps on one A10G GPU at a reported cost of $0.31 wall.

IV. Corpus construction

Because natural same-speaker cross-script Indic data is unavailable at scale, the paper constructs a quality-gated synthetic corpus from multilingual commercial voices.

  • Motivation: Natural Indic corpora lack large-scale same-speaker recordings across scripts, motivating synthetic corpus construction.
  • Synthesis: Eight verified ElevenLabs multilingual voices generate speech in English, Hindi, Telugu, and Tamil.
  • Synthesis: 1600 nominal clips per corpus are created from 50 sentences per language for each voice.
  • Quality gating: 30% of synthesis attempts fail the cosine gate, identifying clips where voice identity drifted across scripts.
  • Quality gating: The gate failures serve as an audit signal and define a concrete degradation floor for cross-script identity preservation.

C. Train / held-out split discipline

The evaluation separates training from content-held-out testing and measures language–identity entanglement through within-script, cross-script, and across-speaker cosine distributions.

  • Train / held-out split discipline: 1118 training pairs and 1043 held-out pairs use the same eight voices but different held-out utterances, isolating content generalisation.Voice-level held-out evaluation with entirely new voices is deferred to v2.
  • Three-distribution evaluation: The within-script distribution compares same-voice clips in the same language across different sentences, serving as an upper bound.The encoder should treat these clips as nearly identical.
  • Three-distribution evaluation: The cross-script distribution compares same-voice clips across languages, directly exposing language-versus-identity leakage.Its distance from within-script similarity is the central test.
  • Three-distribution evaluation: The across-speaker distribution compares different voices in the same language and serves as the noise floor.Each distribution samples 200 random predicate-matching pairs and reports the median cosine.
  • Summary metrics: ∆ measures the within-minus-cross drop, while M measures the cross-minus-floor cushion; desirable encoders minimize ∆ and maximize M.These summaries compare relative spacing within each encoder’s vector space.

VI. Results

Across two held-out corpora, LASE closes the within-versus-cross gap while preserving separation from different speakers; relative spacing matters more than absolute cosine scale.

  • Corpus comparison: Table I reports within-script, cross-script, across-speaker, gap, and margin medians for Western and Indian held-out corpora.The Western corpus contains 1043 pairs, while the Indian corpus contains 1369 pairs.
  • Gap closure: LASE’s ∆ CI includes zero on both corpora, whereas baseline CIs remain strictly above zero.The ECAPA+GRL ablation CI also includes zero but has a wider half-width than LASE.
  • Gap closure: 0.013 is LASE’s Western-corpus gap, 6× smaller than WavLM-SV’s 0.082 and 8× smaller than ECAPA-TDNN’s 0.105.The relative gap closure against WavLM-SV is 84.3%.
  • Interpreting cosine values: LASE’s 0.757 within-script cosine is lower than WavLM-SV’s 0.927 because the encoders occupy different vector spaces with different geometric scales.The meaningful comparison is each encoder’s internal distribution spacing, summarized by ∆ and M.

B. Training dynamics confirm the loss design

Training dynamics match LASE’s loss design: speaker coherence improves while the language classifier remains near four-class chance performance.

  • Speaker objective: L_spk falls from approximately 2.7 initially to approximately 0.5–1.0 by step 900, indicating a voice-coherent embedding geometry.The curves cover 1000 training steps.
  • Language adversary: L_lang stays near ln 4 ≈ 1.386 throughout, so the language classifier does not outperform random prediction.This indicates that the embedding does not carry exploitable language information.
  • Adversarial schedule: λ_t ramps from 0 to 0.1 over the first 700 steps and then holds.Earlier ramps caused SupCon collapse, while later ramps slowed language disentanglement in the tested schedules.

C. Diarisation: matching ECAPA on cross-script recall with 100× less data

On a synthetic multi-speaker code-switching diarisation benchmark, LASE matches ECAPA-TDNN on cross-script recall despite using far less training data, although ECAPA has higher overall ARI.

  • Benchmark: The benchmark contains 50 conversations and 23.7 minutes of synthetic code-switching audio built from 2–4 voices per conversation.Conversations concatenate 6–10 held-out clips with 0.3-second gaps.
  • Metrics and setup: ARI measures clustering agreement with ground-truth speakers, while cross-script recall measures same-speaker clustering across languages.Clustering uses known K and excludes VAD so the evaluation isolates encoder clustering quality.
  • Diarisation results: 0.788 versus 0.789 is LASE’s versus ECAPA-TDNN’s cross-script speaker recall, a statistical tie on the motivating metric.ECAPA-TDNN leads overall ARI, 0.693 versus 0.640.
  • Baseline comparison: Both LASE and ECAPA outperform WavLM-base-plus-sv, whose cross-script recall is 0.604 and ARI is 0.444.The comparison concerns off-the-shelf speaker verification without explicit cross-script training.

D. Generalisation: held-out vs training corpus

LASE generalizes beyond its training pairs and closes the cross-script gap across both accent regimes, while the ablation separates GRL and backbone contributions.

  • Held-out generalisation: 84.3% relative held-out gap closure reduced Δ from 0.082 to 0.013, exceeding the 78.7% training-set closure from 0.099 to 0.021.Both splits were scored against their own WavLM-SV baselines.
  • Ablation: GRL reduced ECAPA’s gap from 0.105 to 0.027 on Western voices and from 0.044 to 0.037 on Indian voices.The relative reductions were 75% and 16%, respectively.
  • Ablation: 0.013 [−.02,.05] versus 0.027 [−.02,.08] on Western voices, and 0.026 [−.04,.08] versus 0.037 [−.03,.10] on Indian voices, with overlapping CIs.WavLM+GRL was directionally better than ECAPA+GRL, but the gap difference was within sampling noise.
  • Ablation: The combination of WavLM and GRL recovered the full effect across both corpora, unlike either GRL-on-ECAPA or WavLM without GRL.The training-loss curves and eval-time gap difference of 0.027 versus 0.013 corroborated this result.
  • Accent dependence: Δ was 0.006 for WavLM-SV on Indian-accented voices versus 0.082 on Western-accented voices speaking the same Indic scripts.The encoder’s baseline gap was therefore strongly accent-conditional.
  • Accent dependence: 70% of Western-voice pairs versus 86% of Indian-voice pairs passed the WavLM-SV ≥0.90 quality gate during corpus construction.The pass rate tracked the same accent-dependent identity-preservation pattern.
  • Overall result: LASE achieved Δ = 0.013 Western and Δ = 0.026 Indian, with bootstrap CIs straddling zero, while its margin advantage ranged from 1.3× to 3.3×.Margin advantages were 2.7× and 3.3× on Western voices, and 2.4× and 1.3× on Indian voices, over WavLM-SV and ECAPA-TDNN respectively.

VII. Discussion

The discussion attributes LASE’s performance to combining an explicit language-invariance objective with a WavLM backbone, while limiting the claims to synthetic, voice-overlapping evaluation and cross-script consistency.

  • VII. Discussion: A tight cross-script same-speaker positive pair leaves identity as the invariant signal, while four distant scripts make the GRL language-classification task easy.The paper presents these as amplifying factors for the adversarial approach.
  • VII. Discussion: WavLM and similar encoders inherit language–identity entanglement from predominantly English VoxCeleb training, especially for phones outside English’s distribution.LASE’s GRL is intended to remove dependence on the subset of phones used by each language.
  • VII. Discussion: LASE is not a drop-in replacement for general speaker verification: ECAPA still wins on overall ARI, while LASE’s advantage is cross-script consistency.The stated deployment wedge is specifically identity consistency across scripts.
  • VII. Discussion: The evaluation uses ElevenLabs Multilingual audio and shares the same eight voices between training and held-out sets, with only sentences held out.Natural-speech transfer and new-voice generalization remain untested.
  • VII. Discussion: The study also leaves IndicWhisper-style encoders, subjective listening validation, and within-clip script-mixed utterances for future experiments.These are identified as explicit v2 follow-ups.

IX. Conclusion

LASE targets language–identity entanglement with a compact adversarial encoder trained on TTS-bootstrapped cross-script pairs. It closes the held-out gap, matches ECAPA on cross-script recall, and uses much less training data.

  • IX. Conclusion: 8–11% of absolute cosine similarity disappears when off-the-shelf encoders represent the same speaker across Indic scripts.The conclusion frames this as language–identity entanglement in speaker encoders.
  • IX. Conclusion: LASE uses a 256-dimensional projection over frozen WavLM-base-plus, supervised contrastive learning, and a gradient-reversal language classifier trained on 1118 synthetic pairs.The pairs came from eight ElevenLabs Multilingual voices.
  • IX. Conclusion: 84% relative held-out gap closure and matched ECAPA-TDNN cross-script speaker recall were achieved with 100× less training data.The conclusion characterizes the recipe as small, cheap, and reproducible.
Loading 2605.00777v1…