PaperlayerView exact revision on arXiv

Source-linked AI summary

Anuvani-LID: Robust and Calibrated Spoken Language Identification for Indian Languages

Stephen Fernandes, Narayan Choudhary

arXiv:2610.20263v1cs.CL

TL;DR

Indian spoken LID must distinguish many related languages in short, noisy, code-mixed audio, while generalist systems have incomplete coverage and struggle on short clips. The paper presents a dedicated model and held-out, multi-axis evaluation; Anuvani-LID leads the tested systems, particularly on short, noisy, and all-23-language conditions, and is well calibrated.

  • Problem

    Indian spoken LID covers related languages and challenging audio, while tested generalist systems lack some language classes and struggle on short inputs.

    Source: §1 · p. 2

  • Method

    The paper presents Anuvani-LID, a dedicated LID model, and evaluates it through a held-out, multi-axis benchmark and controlled fine-tuning studies.

    Source: §1 · p. 2, §5.2 · p. 5

  • Results

    Anuvani-LID leads the tested systems, especially on short, noisy, and all-23-language conditions, and is well calibrated.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that reliable Indian LID still requires a robust, dedicated, calibrated model, with code-switching, dialect continua, and unseen relatives remaining a frontier.

    Source: §9 · p. 15

    At 0.25 seconds, Anuvani-LID reaches 0.24–0.35 macro-F1 in seven of eight conditions, and full-scale Stage VI training was not repeated with other seeds.

    Source: Limitations · p. 15

Abstract

from arXiv · show

Every multilingual speech pipeline begins with a language decision, an error there propagates silently through recognition, translation and routing. In India it spans 22 scheduled languages across four language families, heard over short, noisy, narrowband and code-mixed audio. Foundational speech models increasingly present language identification (LID) as an inherent latent, generalized capability. Autoregressive decoders in speech models emit a special language ID token, and modern LLMs with audio capabilities can name the language when prompted. We test this assumption on identical clips and find that whisper and similar speech models has tokens for only 15 of the 23 languages, on one second inputs Gemini 3.5 Flash reaches 0.52 macro-F1 and GPT-audio 0.14. We present Anuvani LID, a dedicated 115M parameter LID conformer model trained on 54,643 hours of Indian speech spanning 22 Indian languages and Indian accented English. Along with designed and controlled studies on 102 condition benchmark with speaker and corpus held out tests. Across 37 conditions, it reaches 0.914 mean macro-F1, 5.2 points above Gemini 3.5 Flash and ahead of 14 other systems. It is 24 points ahead of Gemini at one second, scores 0.897 on all 23 languages where no other system exceeds 0.713, and is calibrated to an ECE of 0.008. Reliable Indian LID still requires a robust, dedicated and calibrated model.

1 Introduction

The introduction argues that Indian spoken LID remains difficult for generalist systems because language coverage is incomplete and short, challenging audio exposes errors. It presents a held-out benchmark and controlled studies to evaluate a dedicated, calibrated model.

1 Introduction

  • At one second, Gemini 3.5 Flash reaches 0.52 macro-F1 and GPT-audio 0.14, while most speech models fail below one second.Whisper has language-ID tokens for only 15 of the 23 classes.Source: §1 · p. 2
  • A 102-condition, 318,861-utterance benchmark with speaker- and corpus-held-out tiers addresses saturation in clean, in-domain tests.The pilot study found every head reached about 98% macro-averaged accuracy, within 1 pp of the others.Source: §1 · p. 2
  • The study compares 15 open and commercial systems on identical clips in SUITE-42.SUITE-102 includes speaker- and corpus-held-out tiers.Source: §1 · p. 2
  • The paper conducts a staged, controlled study of fine-tuning a self-supervised Conformer for LID.It also reports linguistic error analysis and finds that 12 of 17 blocks suffice.Source: §1 · p. 2
  • Anuvani-LID leads the 15 tested systems, especially on short, noisy, and all-23-language conditions, and is well calibrated.The introduction frames calibrated scores as useful for routing and rejection, which hard-label APIs often lack.Source: §1 · p. 2

2 The Linguistic Landscape of Indian Spoken LID

Indian spoken LID spans languages with uneven resources, close linguistic relationships, and substantial within-class variation. Speech lacks script cues, leaving sparse phonetic, phonotactic, and prosodic evidence that can be hard to capture in short clips.

2 The Linguistic Landscape of Indian Spoken LID

  • The 23 classes comprise 22 scheduled languages and Indian-accented English, while training hours and speaker populations vary widely.The passage identifies the labels as administrative and notes that Census categories group multiple mother tongues.Source: §2 · pp. 2–3, Table 1 · p. 3
  • Shared features across language families and close relatives make genealogy an incomplete guide to acoustic similarity.Hindi and Urdu share colloquial Hindustani; the passage also notes close vocabulary relationships among several other language pairs.Source: §2 · p. 3
  • Because script is absent from speech, language evidence must come from corpus metadata and acoustic cues.The cited cues include retroflexion, laryngeal contrasts, vowel length, and lexical tone.Source: §2 · pp. 3–4
  • Many distinguishing cues are sparse, and a one-second clip may contain none of them.The passage connects this sparsity with duration being the dominant difficulty axis.Source: §2 · p. 4
  • Indian-accented English must accommodate accent variation that also carries traces of speakers’ mother tongues.Indian English is the 23rd class, and all English training and test audio is Indian-accented.Source: Table 1 · p. 3, §2 · p. 4
  • Code-switched speech complicates utterance-level labeling because embedded segments can be very short.More than half of embedded segments in the cited study last only 200–400 ms, making the matrix language a natural utterance label.Source: §2 · p. 4

3 Related Work

Prior spoken LID work spans conventional speaker-recognition methods and self-supervised speech encoders, while Indian-language studies have highlighted cross-corpus and unseen-speaker challenges. This paper builds on Indian-audio pretraining and differs in scale, class coverage, held-out evaluation, and comparison with commercial audio LLMs.

3 Related Work

  • Spoken LID progressed from i-vectors and x-vectors to ECAPA-TDNN and self-supervised encoders, including fine-tuned wav2vec 2.0-style models.Prior work also includes fine-tuning an SSL Conformer; NIST LRE uses cluster-aware Cavg evaluation.Source: §3 · p. 4
  • Indian-language LID studies report substantial cross-corpus degradation and lower accuracy on unseen speakers.The cited cross-corpus EER rises from 11.8% in-corpus to 43.3% cross-corpus; IndicSUPERB accuracy falls from 98.2% on seen to 90.8% on unseen speakers.Source: §3 · p. 4
  • This work differs through 54.6k hours of training, all 23 spoken-language classes, held-out multi-axis evaluation, and commercial audio-LLM comparisons.It builds on MahaDhwani, pretrained on 279k hours of Indian audio.Source: §3 · p. 4

4 SUITE-102: A De-saturated Benchmark

SUITE-102 evaluates LID across held-out speakers, corpora, varieties, durations, and acoustic conditions. SUITE-42 compares systems on identical clips, using a balanced subset because commercial APIs are costly to run across the full suite.

4 SUITE-102: A De-saturated Benchmark

  • SUITE-102’s headline tier tests 23 classes on 300 spontaneous utterances per language from 1,308 speakers absent from training.Cross-corpus tiers use held-out splits, and the suite includes variety probes and clips expanded into crops from 0.25–10 seconds.Source: §4 · p. 5
  • The suite probes noise, reverberation, telephony, babble, confusable clusters, and open-set trials, and was frozen before the final training run.Open-set trials include music, eight non-Indian languages, and Chhattisgarhi.Source: §4 · p. 5
  • SUITE-42 samples 42 conditions, including a 37-condition, 47,730-clip panel across ten languages and a five-condition, 17,229-clip all-23 panel.Every system receives identical audio; Anuvani-LID predicts over all 23 classes and is also reported matched to ten candidates.Source: §4 · p. 5
  • Macro-F1 is the primary metric, supplemented by top-1 accuracy, Cavg, EER, 15-bin ECE, and open-set AUROC.Intervals use bootstrap resampling, and paired items are tested with McNemar’s test under Holm or Bonferroni correction.Source: §4 · p. 5

5 Model and Staged Research Programme

Anuvani-LID fine-tunes a self-supervised Conformer using learned layer aggregation, attentive pooling, and focal loss. A six-stage controlled programme selects training choices against held-out conditions, culminating in a checkpoint soup that improves robustness and calibration over the final checkpoint.

5.1 Model

  • Anuvani-LID fine-tunes all 17 MahaDhwani Conformer blocks, learns their aggregation weights, then applies attentive statistics pooling and a 23-way classifier.The model has 115.6M parameters and uses focal loss with γ=2 and label smoothing of 0.1.Source: §5.1 · p. 5, Figure 3 · p. 6

5.2 A staged research programme

  • At Stage I, attentive pooling with focal loss and full fine-tuning reached 0.840 cross-corpus macro-F1, exceeding ECAPA-TDNN variants despite 23M extra parameters.Focal loss yielded ECE 0.023; frozen-encoder and partial fine-tuning regimes trailed full fine-tuning, while layer aggregation improved top-1 correctness by 1.1%.Source: §5.2 · pp. 6–7, Figure 4 · p. 7
  • At Stage II, cross-corpus macro-F1 peaked at 0.919 with sampling temperature T=2, while Dogri and Konkani mean F1 rose from 0.742 at T=1 to 0.930.The eight languages above 2,000 hours slipped from 0.949 to 0.935, making temperature a head–tail trade-off; in-domain macro-F1 was 0.946 at T=2.Source: §5.2 · pp. 7–8, Figure 5 · p. 7
  • Two epochs at T=2 raised pooled cross-corpus macro-F1 from 0.919 to 0.931 and improved every reported axis and resource tier.Focal loss complements sampling by weighting hard examples: it raised cross-corpus macro-F1 by 0.86 points over angular-margin loss and cut ECE from 0.788 to 0.023; learned layer aggregation raised in-domain macro-F1 from 0.958 to 0.967.Source: §5.2 · p. 8
  • Stage III perturbations mainly improved the audio conditions they simulate; the combined stack gained 10.7 points at 0.5 seconds and cost 0.4 points on clean speech.Multi-talker babble added 11.6 points on seven babble conditions, from 0.804 to 0.920 macro-F1, when added to the other four perturbations.Source: §5.2 · pp. 8–9, Figure 6 · p. 8
  • Duration-based self-paced training raised mean macro-F1 across five corpora to 0.52 at 0.5 seconds and 0.26 at 0.25 seconds, versus 0.42–0.43 and 0.17–0.18 without scheduling.The final recipe retained bimodal short-crop exposure while keeping long segments in every batch; the adopted confusability curriculum targeted Indo-Aryan errors while preserving the tuned sampling mix and language floors.Source: §5.2 · p. 10
  • Stage V selected a 3×10^-4 learning rate and T=1.75 with English and Hindi sampling floors; the 4,000-hour proxy reached 0.941 development accuracy after 25k steps.The full-scale Stage VI run trained on 54,643 hours for 425k steps; its checkpoint trajectory peaked at different times across condition families, motivating trajectory averaging.Source: §5.2 · pp. 10–11
  • 0.8453 SUITE-102 macro-F1: a seven-checkpoint soup was the best of 31 candidates and became the released model.Against the final checkpoint, it raises telephony macro-F1 from 0.898 to 0.923 and lowers all-23-test raw ECE from 0.077 to 0.054, with no condition family falling by more than 0.5 points.Source: §5.2 · pp. 11–12, Figure 8 · p. 11

6 Comparison with Open and Commercial Systems

Anuvani-LID leads across most evaluated conditions, with its clearest margins on short and all-23-language inputs. Gemini performs better on several long read-speech conditions, while Anuvani-LID’s held-out all-23 predictions are well calibrated.

Overall

  • 0.914 mean macro-F1 on 37 SUITE-42 core conditions puts Anuvani-LID 5.2 points above Gemini 3.5 Flash and 8.8 points above the best open model.The paired differences exclude zero (McNemar p < 10^-100 over 47,730 clips).Source: §6 · p. 12

Where the margin comes from

  • Anuvani-LID has the best mean in eight of nine domains, with its largest Gemini margins on short utterances, babble, 0 dB noise, and reverberation.It is significantly behind Gemini on six long read-speech conditions, including the five full-length FLEURS slices and MUCS.Source: §6 · p. 12, Figure 9 · p. 13

Short utterances

  • At one second, Anuvani-LID reaches 0.764 macro-F1 on FLEURS and 0.854 in-domain, 24 points above Gemini in both cases.At 0.5 seconds it reaches 0.545, versus 0.439 for Sarvam and 0.231 for Gemini.Source: §6 · p. 12

All 23 languages

  • 0.897 average macro-F1 on the all-23 panel exceeds Voxlect Indic-Whisper (0.713), Gemini (0.710), and GPT-audio (0.384).Anuvani-LID exceeds 0.90 F1 on 21 languages; a rival is marginally ahead only on Tamil.Source: §6 · p. 12

The full suite against open models

  • Across 73 SUITE-102 conditions where an open model covers at least eight core languages, Anuvani-LID scores highest on 71, with a mean margin of 12.8 points over the best open model.The exceptions are FLEURS telephony and IndicTTS.Source: §6 · p. 12

Calibration, open set and selection

  • Temperature scaling lowers held-out all-23 ECE from 0.054 to 0.008 using a single temperature fitted on the development set.The unselected final EMA checkpoint scores 0.912 core macro-F1 and 0.889 all-23, so selecting the soup on SUITE-102 changes no ranking.Source: §6 · pp. 12–13

7 Linguistic Error Analysis

Anuvani-LID’s residual errors are structured rather than evenly distributed: they often remain within language families or reflect contact and first-language influence. Confusions include close relatives, neighboring languages, and Indian English attributed to scheduled languages.

Residual errors

  • 61% of all-23 test errors stay within the reference language’s family, and 73% originate in Indo-Aryan languages.The confusion analysis is on the speaker-disjoint all-23 test.Source: §7 · p. 13

Sisters within a subgroup

  • The largest within-subgroup confusions include Hindi→Maithili (4.7%), Maithili→Hindi (2.0%), and Urdu→Hindi (4.0%).Other examples include Konkani→Marathi and Odia→Assamese or Maithili; Hindi is the hardest class at F1 0.84.Source: §7 · p. 13

Neighbours across family lines

  • Several cross-family confusions stay within geographic contact regions, including Assamese→Bodo (3.0%) and Manipuri→Bodo (2.7%).The authors note that shared recording regions may also contribute to these patterns.Source: §7 · p. 13

L1 transfer in Indian English

  • Indian English is confused mainly with Hindi (2.7%), Santali (2.0%), and Marathi (1.7%), matching L1-conditioned phonology.On Svarah, 40% of accented-English utterances are attributed to a scheduled language, spread over many classes.Source: §7 · p. 13

Hindi varieties

  • On Hindi-variety probes, 19% of Lahaja dialectal Hindi is labelled Urdu, while HiACC Hinglish is labelled Urdu 28% and English 23%.The authors say single-language probes reward a prior toward the probed class; colloquial Hindi and Urdu share one form, and code-mixed speech calls for frame-level language diarization.Source: §7 · pp. 13–14

8 Where Is Language Identity Encoded?

The authors probe where language identity is encoded using block-output probes, learned aggregation weights, and encoder truncation. Their findings place useful language information in the mid-to-upper stack and show that aggregation and truncation affect robustness and performance.

Findings

  • In the pre-trained encoder, probe macro-F1 peaks at about 0.56 in blocks 9–11, then collapses to 0.13 at block 17, whose representations serve the pre-training objective.Fine-tuning moves the peak to block 11 (0.69) at 1,000 h.Source: §8 · p. 14
  • At full scale, block 17 receives 30% of learned aggregation weight and blocks 10–16 receive most of the remainder; no single-block knockout costs more than 0.08 macro-F1.Bypassing block 7 costs a last-block model 0.34 macro-F1.Source: §8 · p. 14, Figure 13 · p. 15
  • Erasing the linearly decodable rank-3 corpus-identity subspace changes LID macro-F1 by only +0.1 points.Corpus identity remains linearly decodable at the top (0.90; majority 0.25), but the decision does not rest on that subspace.Source: §8 · p. 14
  • Truncation saturates at 12 blocks: 0.848 macro-F1 with 84M parameters versus 0.840 with all 17 blocks and 116M.At four epochs, the corresponding scores are 0.844 and 0.835 with two seeds.Source: §8 · pp. 14–15, Figure 13 · p. 15

9 Conclusion

The conclusion is that Indian-language LID is not yet a solved byproduct of model scale: tested generalist systems have class-coverage, short-input, or calibration weaknesses. The authors argue for a dedicated, robust, calibrated model and identify code-switching, dialect continua, and unseen relatives as future challenges.

Conclusion

  • For India’s 22 scheduled languages and Indian English, tested generalist systems lack classes, collapse on short inputs, or return uncalibrated labels.The authors conclude that a dedicated 116M-parameter model designed on held-out, multi-axis conditions does better on each count.Source: §9 · p. 15

Limitations

The paper bounds its claims by utterance-level labeling, a 23-class label set, evaluation constraints, and limited replication at full scale.

Label granularity

  • Single-label utterance classification is ill-posed for code-switched speech, and HiACC and Lahaja probes show penalties on mixed and dialectal Hindi.Frame-level language diarization is identified as the appropriate formulation for code-switching.Source: Limitations · p. 15

Coverage

  • The label set covers 22 scheduled languages and Indian English, but excludes other languages; rejecting close relatives such as Chhattisgarhi is harder than rejecting unrelated languages.Source: Limitations · p. 15

Evaluation

  • Commercial APIs return hard labels, so calibration and Cavg comparisons are unavailable, and their behavior may change between versions.Kathbath shares four of 456 speakers with training and is reported only as an in-domain comparator.Source: Limitations · p. 15

Variance and scale

  • Stages I–V used single runs and Stage VI was trained once, leaving small within-stage differences provisional and full-scale results unreplicated across seeds.At 0.25 s, Anuvani-LID reaches 0.24–0.35 macro-F1 in seven of eight conditions; no open model exceeds 0.18 where scored.Source: Limitations · p. 15

Ethics Statement

The paper uses licensed public-corpus data without private user audio, describes limits on identity-related interpretation and gender analysis, and specifies its benchmark and model-comparison setup.

Ethics Statement

  • The study uses published licensed corpora and no private user audio; its production-mix condition is a held-out sample of public training corpora in deployment proportions.Source: Ethics Statement · p. 15
  • LID output is not recommended as a proxy for ethnicity, religion, or nationality; Hindi and Urdu share a spoken register, so labels should not be read as speaker identity.Source: Ethics Statement · p. 16
  • About one third of training speech with recorded gender is female, but gender-sliced performance was not evaluated and is flagged as future work.Source: Ethics Statement · p. 16

A Benchmark composition

  • SUITE-102 averages macro-F1 equally across 102 conditions, including speaker-held-out, split-held-out, corpus-held-out, and clip-held-out tiers.Speaker-held-out tiers share no speakers with training; corpus-held-out probes use corpora absent from training.Source: A Benchmark composition · p. 23

B Research programme and Stage VI settings

C Compared systems: versions and dates

  • Systems received identical core and all-23 clips, while prompted audio LLMs and Azure received candidate lists and open models used native inventories.All API queries were made between 16 and 21 July 2026.Source: C Compared systems: versions and dates · p. 23

D Additional results

Additional results compare top-1 accuracy and macro-F1 across systems, identify settings where competitors perform better, and report variants and benchmark scope.

  • 0.895 mean top-1 accuracy over the 37 core conditions for Anuvani-LID, versus 0.858 for Gemini 3.5 Flash and 0.865 for Sarvam Saaras v3.Anuvani-LID's matched score is 0.915.Source: D Additional results · p. 23
  • Sarvam's top-1 accuracy exceeds Gemini's while its macro-F1 is lower, indicating errors concentrated in a few languages.Source: D Additional results · p. 23
  • MMS-LID-512 and MMS-LID-1024 score 0.808 and 0.801 mean macro-F1 on 37 core conditions, with paired deficits versus Anuvani-LID of 10.6 and 11.3 points.Both cover 21 of 23 classes; enlarging the MMS-LID inventory beyond 126 languages does not help this 23-way Indic set.Source: D Additional results · p. 23
  • Gemini 3.5 Flash is significantly ahead on five full-length FLEURS slices and MUCS read speech, while those sources reverse the ranking when cropped to one second.On clean FLEURS, the reported macro-F1 comparison is 0.961 against 0.977; on MUCS it is 0.974 against 0.986.Source: D Additional results · p. 23
  • SUITE-102 spans 102 conditions, including duration, noise, reverberation, telephony, babble, confusable-cluster, and open-set probes.The benchmark contains 318,861 utterances, including 1,900 out-of-set negatives across three open-set conditions.Source: D Additional results · p. 24, Table 4 · p. 24
  • Table 8 reports macro-F1 for Anuvani-LID and commercial APIs across 42 SUITE-42 conditions, with a matched row restricting Anuvani-LID to the prompted APIs' candidate set.Source: Table 8 · p. 25

Related papers

Loading 2610.20263v1…