Convex Low-resource Accent-Robust Language Detection in Speech Recognition

1. Introduction

• Diverse accents and dialects can trigger language misidentification in ASR, while sparse intonation annotation leaves regional speech underrepresented in transcription datasets.The introduction gives Singlish, whose intonation and prosody can lead to mistranscription as Bahasa or Tamil, as an example.Source: §1 · p. 1
• Scarce, costly-to-curate audio limits training data, while speaker variation, codeswitching, and disfluencies compound distribution mismatch and downstream dialogue errors.The paper frames sample-efficient algorithms as one approach to making progress with limited speech data.Source: §1 · pp. 1–2
• CLD applies convex optimization to low-resource spoken-language classification, using an ADMM solver implemented in JAX for fast training and inference.The paper claims polynomial-time global optimality and positions CLD as a practical convex-optimization application for language identification in speech dialogue systems.Source: §1 · p. 2
• The convex reformulation yields exact logit-Lipschitz constants and certified margin stability against hidden-feature perturbations, with a computable, data-dependent radius for label invariance.The guarantee is intended to ensure predictions remain stable within that radius.Source: §1 · p. 2
• 97-98% accuracy in low-resource regimes on Whisper Large v3 and MMS-1B; CLD remains performant with fewer than 100 training samples across five languages and 24 sub-dialects.The paper reports that CLD consistently outperforms competitors in the large-model experiments.Source: §1 · p. 2
• The authors provide an installable JAX package and open-source code as a reproducible, deployable plug-in module for robust speech-driven tools.The introduction connects this release to global inclusivity and equitable access.Source: §1 · p. 2

Multilingual Tasks

• Multilingual ASR models such as Whisper cover over 99 languages, yet low-resource performance drops; transfer learning uses high-resource speech paired with target-language text but depends on linguistic basis similarity and risks catastrophic forgetting.The cited passage reports that models generally perform best on English and that forgetting can deteriorate overall performance.Source: §2 · p. 2

Low-resource Environments

• Even English and Mandarin dialects challenge ASR; prior work models prosody or acoustic context and examines low-resource challenges, but these methods depend on large fine-tuning datasets and learning rates ten times smaller than standard supervised fine-tuning.The cited work includes prosody-assisted systems, bidirectional LSTMs, and research on spoken-dialogue models.Source: §2 · pp. 2–3

Certified Robustness and Lipschitz analysis

• Prior robustness work estimates local Lipschitz behavior of trained non-convex networks or studies speech perturbations, interference, and network motifs; this work instead derives a data-dependent variation-norm bound and hidden-feature margin certificate.The bound directly limits logit perturbations rather than estimating a black-box local constant after training.Source: §2 · p. 3

Convex Programs

• Convex reformulations offer polynomial-time convergence to global optima, but prior work focused on theory or small image benchmarks; CRONOS demonstrated GPT-2-scale binary language classification, while this work targets multiclass speech representations in data-scarce settings.The paper frames its contribution as scaling convex training to high-dimensional speech representations for spoken dialogue systems.Source: §2 · p. 3

3.1. Convex Two-Layer ReLU Networks

• The convex reformulation represents ReLU activation patterns as a convex program, with equivalence to non-convex training under width and all-pattern conditions; sampled patterns may differ marginally.The paper reports that this discrepancy is negligible in practice (Kim & Pilanci, 2024).Source: §3.1 · pp. 3–4

3.2. Convex Language Detection Algorithm

• CLD extracts hidden representations from the ASR encoder and trains a lightweight language-detection head using a multi-GPU, batched CRONOS implementation in JAX.The implementation targets scale, speed, and robustness for high-dimensional, sensitive multiclass speech data.Source: §3.2 · p. 4
• Sub-500ms inference uses masked mean pooling over encoder representations, predicts a language token with one lightweight forward pass, and supplies it to the decoder to condition transcription.The online pipeline augments an encoder–decoder ASR system such as Whisper.Source: §3.2 · p. 4, Figure 1 · p. 4

4. Theoretical Analysis

• The convex-trained CLD detection head is Lipschitz-stable in encoder feature space, so bounded feature perturbations degrade its one-vs-rest margin at most linearly and sufficiently large margins certify label invariance.The section presents the complete derivation in Appendix A.Source: §4 · pp. 4–5

4.1. Margin Stability in Hidden Features

• For hidden-feature perturbation δ, the predicted class is unchanged if ||δ||2 < mar(h,y)/(2||f||var); if encoder E is LE-Lipschitz, it is preserved when ||η||2 < mar(E(x),y)/(2||f||varLE).Source: §4.1 · p. 5

4.2. Robustness Certificates from the Convex Program

• A feasible convex solution gives an explicit upper bound on ||f||var from optimization-variable block or Frobenius norms, enabling learned weights to yield data-dependent Lipschitz and margin-radius certificates.For an encoded utterance, perturbations below the resulting feature-space radius leave the predicted language unchanged.Source: §4.2 · p. 5
• The primary certificate is in feature space; an audio-space radius rh/LE requires a reliable encoder Lipschitz bound, which may be highly pessimistic for deep Transformers.The paper treats end-to-end audio certificates as conservative diagnostics rather than tight robustness claims.Source: §4.2 · p. 6
• Any feasible convex solution also matches training predictions with a vector-valued two-layer ReLU network of at most 2PK hidden units, supporting the variation-norm certificate.The convex head has a finite two-layer ReLU representation with the same objective value, up to negligible approximation from activation-pattern sampling.Source: §4.1 · p. 5, §4.2 · p. 6

5. Experiments

• Evaluation covered WER, CER, language-detection accuracy, training time, and computational efficiency, comparing Whisper-Small, Whisper-Large-V3, and MMS-1B with default detection, neural, SVM, Kernel SVM, and KNN classifiers.The neural baseline used the same encoder embeddings and output labels as CLD, with a 256-dimensional hidden layer, ReLU, dropout, and an output layer.Source: §5 · p. 6

5.1. Speech Datasets

• The experiments used Common Voice, Singapore’s National Speech Corpus, and Lahaja, with audio normalization and augmentation; binary tests varied 100–10 000 samples across five dialects per language, while multiclass tests used 16 000 samples across five languages and 24 accents.Lahaja contributes 12.5 hours of Hindi from 132 speakers across 83 Indian districts; the multiclass data used an 80-10-10 train-test-validation split.Source: §5.1 · p. 6

5.2. Main Results and Discussion

• CLD maintained 96.94–99.14% detection accuracy across sample sizes and achieved the lowest WER, 21.62, at 10 000 samples.The vanilla NN and fine-tuned Whisper-Small showed improving WER and detection accuracy as training data increased, while CLD was consistent across sample sizes.Source: Figure 2 · p. 7, §5.2 · p. 7
• CLD trained in 64.45 seconds—about 7.7% of vanilla NN runtime—and used 13x fewer TFLOPs.The authors attribute this efficiency to a convex formulation solved via ADMM in JAX, with parallelizable updates and a unique global optimum.Source: Table 1 · p. 7, §5.2 · p. 7
• At 500 samples per language, CLD reached 88.73% accuracy on Min Dong Chinese and exceeded 94% on other dialects, while the NN misclassified 88.88% of Chinese samples as English.The NN achieved just 8.78% accuracy on Mainland Chinese and 8.84% on Taiwanese; CLD’s results were uniformly high across accents.Source: §5.2 · pp. 7–8, Table 2 · p. 8
• On the five-language multiclass task, CLD achieves the best performance across evaluation metrics, with MMS-1B gains of up to 44.78% in detection accuracy and a 12.74% WER decrease versus baseline.The study also reports accuracy increases and WER decreases of up to 29.21% when augmenting larger pretrained models.Source: Table 3 · p. 8, §5.2 · p. 8
• On Whisper-Small, CLD showed strong diagonal dominance across five languages, with its lowest accuracy at 95% for English; the vanilla NN reached only 44% on Chinese.The NN also confused Chinese as Indonesian in 34% of cases and Malay as Indonesian in 23%.Source: §5.2 · p. 8, Figure 3 · p. 9
• In a five-participant Singapore-English and ten-participant southeastern-China Mandarin case study, CLD reduced wrong-language outputs and word errors, though the study was not statistically powered for population-level evaluation.Default detection sometimes routed accented English or Mandarin into the wrong decoding language, producing unusable cross-lingual transcripts.Source: §5.2 · p. 9

6. Conclusion

• CLD uses a convex reformulation to support certified robustness and sample efficiency while maintaining consistent, high accuracy across dataset sizes and mitigating English-centric ASR failures on diverse low-resource dialects.The paper characterizes the framework as fast, lightweight, and theoretically grounded.Source: §6 · p. 9
• Future work will integrate CLD more deeply into traditional ASR for end-to-end continuous representation learning and investigate the framework in multimodal agentic models.Source: §6 · p. 9
• A differentiable CLD pipeline could back-propagate through the convex program by unrolling ADMM iterations or differentiating implicitly through the KKT conditions of Equation (2).This could enable training encoders whose latent representations are optimized for convex language separability.Source: §6 · p. 9

• The method aims to democratize spoken dialogue models by improving access for users whose input dialects are not well understood and encouraging tools for diverse speech variations.Source: Impact statement · p. 10

A. Proof of Main Results and certificates for margin stability

• The section links the convex program in Eq. 2 with a two-layer ReLU representation and computable certificates that translate into certified radii.Source: A. Proof of Main Results and… · p. 14

A.1. Proof of Lemma 4.3; Proof of Theorem 4.4

• The Lemma 4.3 variation-norm bound supports Theorem 4.4 by bounding each difference and yielding the stated result.The proof maximizes over k and takes an infimum over representations; the subsequent proof applies Lemma 4.3 to each difference.Source: §A.1 · p. 14, Proof of Theorem 4.4 · p. 14

A.2. Activation patterns and pattern cones

• Activation-pattern cones characterize vectors whose training activation pattern matches a fixed pattern, apart from measure-zero ties.The appendix defines training activation patterns and associated cones, then gives the equivalence between cone membership and matching the pattern.Source: §A.2 · p. 14

A.3. From the convex program to a two-layer ReLU

• The sampled-pattern convex model maps to a two-layer ReLU representation, with multi-class weights arranged in matrices and a block ℓ2,1 norm option.The columns of the weight matrices correspond to class-specific vectors.Source: §A.3 · p. 14

A.4. Variation norm and a computable certificate; Proof of Theorem 4.7; A.6. Details for the logit Lipschitz bound

• Variation-norm bounds control logit changes and yield computable Lipschitz certificates, including an end-to-end bound when the encoder is Lipschitz.The logit difference is bounded in the infinity norm by the variation norm times the input distance; an L_E-Lipschitz encoder replaces that distance with L_E times the original-input distance.Source: §A.4 · p. 14, Proof of Theorem 4.7 · p. 15, §A.6 · p. 15

A.5. AM–GM link to the nonconvex ℓ2 2 penalty

• Any solution of Eq. 2 yields the stated certificate through the AM–GM link, and larger β tightens the bound linearly.The passages do not provide the certificate’s formula or numerical values.Source: §A.5 · p. 15

B. Related Work Continues

• Low-resource speech work documents training-data scarcity through community recording partnerships, while noting ethical questions and slow annotation of global low-resource languages.The cited work identifies data scarcity as a dominant challenge and describes remote participant recordings and collaboration across ASR and HCI.Source: B. Related Work Continues · p. 15
• Unlike empirical defenses and black-box certification, CLD derives a constructive margin-stability certificate from its convex reformulation; prior audio robustness work typically targets environmental noise, not dialect shifts.The passage describes a variation-norm certificate that directly bounds the detection head’s Lipschitz constant.Source: B. Related Work Continues · p. 15
• CLD uses batched multi-GPU ADMM in JAX for the convex reformulation, with an implementation intended to integrate into ASR pipelines and encoder–decoder architectures.The passage also presents this design as aligned with recent JAX-based convex and LLM systems and as a path toward scaling to larger encoders, cloud TPUs, and broader applications.Source: B. Related Work Continues · p. 16

C. Alternating Direction Method of Multipliers

• ADMM breaks convex minimization into smaller local subproblems linked by Lagrange multipliers and a quadratic penalty, making objectives separable across variable subsets.The framework is designed to handle complex or non-smooth regularization terms.Source: C. Alternating Direction Method of… · p. 16
• For CLD’s convex reformulation of a two-layer ReLU network, ADMM enables batched multi-GPU optimization in JAX and isolates the non-smooth term in a proximal-update subproblem.The method addresses the optimization challenge posed by high-dimensional ASR hidden features.Source: C. Alternating Direction Method of… · p. 16
• Solving CLD’s convex program with ADMM is claimed to converge to the unique global optimum, avoiding brittle learning-rate dependence and local minima associated with standard MLP training.The passage contrasts this guarantee with traditional neural-network training.Source: C. Alternating Direction Method of… · p. 16
• CLD’s ADMM training requires 13x fewer TFLOPs than the standard vanilla-NN and reduces training time; its use for large-scale speech-system reformulations is presented as novel.The authors contrast ADMM’s established use in control and optimization with its practical application as a core training method in spoken dialogue systems.Source: C. Alternating Direction Method of… · p. 16

• Figure D.1 compares character error rates for Whisper-small default, Whisper-small fine-tuned, vanilla-NN, and CLD at training sizes of 100, 500, 1000, and 10 000.Source: Figure D.1 · p. 17
• Figure D.2 presents Whisper-small language-classification performance for WSP-SFT and WSP across English, Chinese, Indonesian, Malaysian, Hindi, and other predicted languages.Source: Figure D.2 · p. 17, D. Extended Experimental Results and… · p. 17

D.1. Classification Accuracy per Sample Size Ablation

• Multi-dialect EN–ZH accuracy is tabulated for WSP, WSP-SFT, vanilla-NN, and CLD at three sample sizes.The conditions are 100 samples per language and 20 per dialect; 1,000 and 200; and 10,000 and 2,000, respectively.Source: Table D.1 · p. 18, Table D.2 · p. 18, Table D.3 · p. 18

D.2. Impact of Hyperparameter Tuning on Vanilla NN Performance

• The appendix reports hyperparameter-tuned vanilla-NN accuracy across sample sizes and metrics against WSP, WSP-L, and MMS-1B.The tables cover 100, 500, 1,000, and 10,000 samples; the model comparison includes WER, CER, and detection accuracy, with accuracy changes from non-tuned models shown in parentheses.Source: §D.2 · p. 19, Table D.4 · p. 19, Table D.5 · p. 19

D.3. Low-Resource Dialects in High-Resource Languages.

• Whisper accuracy was 100% on Midwestern English and 61.4% on Malaysian-accented English, despite both being English.The paper presents this gap as an example of degraded performance on dialectal speech.Source: §D.3 · p. 19, Table E.6 · p. 19

E. Language classification accuracy of Baseline Whisper ASR

• Table E.6 gives Whisper-small’s default language-detection accuracy by language and accent.The supplied passage identifies the table’s scope but does not provide its full cell values.Source: Table E.6 · p. 19

F. Qualitative Human Case Study

• The illustrative human case study compares Whisper-small’s Default, vanilla-NN, and CLD classifiers.It involved five participants in Singapore for English and ten participants in southeastern China for Mandarin.Source: Table F.7 · p. 21

G. Hyperparameter Selection and Hardware Setup

• CLD uses the subroutines’ suggested defaults, with neuron count adjusted for binary versus multiclass classification.Binary settings are rank 20, β = 10^-3, ρ = 10^-4, γ-ratio 1, six ADMM iterations, 32 PCG iterations, and 10 neurons; multiclass uses 32 neurons with the other settings unchanged.Source: G. Hyperparameter Selection and… · p. 21
• The NN and linear SVM baselines use validation-selected hyperparameters, with standardized embeddings for the SVM.The NN uses AdamW and grid searches learning rate, weight decay, and epochs; LinearSVC runs up to 5,000 iterations and grid-searches C.Source: G. Hyperparameter Selection and… · p. 21
• Kernel SVM and k-NN settings are validation-selected, and experiments ran on four 40-GB NVIDIA A100-SXM4 GPUs.Both baselines standardize embeddings; the kernel SVM searches kernel type, C, and γ, while k-NN searches k, distance metric, and weighting scheme. Models use an 80-10-10 train-test-validation split.Source: G. Hyperparameter Selection and… · p. 21