PaperlayerView exact revision on arXiv

Source-linked AI summary

Convex Low-resource Accent-Robust Language Detection in Speech Recognition

Miria Feng, William Tan, Mert Pilanci

arXiv:2605.23235v1cs.LG

TL;DR

Spoken dialogue systems often misidentify languages in diverse accented and dialectal speech, creating downstream ASR failures under low-resource conditions. CLD applies convex language detection to ASR encoder features, with theoretical stability guarantees and efficient optimization; it achieves consistent high accuracy across sample sizes and dialects.

  • Problem

    Spoken dialogue systems frequently misidentify input languages in accented and dialectal speech, causing cascading downstream errors and mistranscription, especially under mismatched training and real-world distributions.

  • Method

    CLD applies a lightweight convex detection head to ASR encoder hidden representations, trained with a multi-GPU batched JAX implementation and analyzed through margin-stability guarantees.

  • Results

    96.94% minimum detection accuracy with 10 000 samples and 99.14% maximum with 1000 samples demonstrate consistent performance across low-resource sample sizes.

  • Takeaways & Limitations

    CLD mitigated English-centric ASR failure modes and maintained high accuracy across dataset sizes and diverse low-resource dialectal speech conditions.

  • Takeaways & Limitations

    Future work must integrate CLD more deeply with ASR models and make the full pipeline differentiable for end-to-end representation learning.

Abstract

from arXiv · show

Globalization and multiculturalism continue to produce increasingly diverse speech varieties. Yet current spoken dialogue systems frequently fail on under-represented dialects and accents, often misidentifying the input language and causing cascading failures in downstream dialogue tasks. Addressing this dialectal variance under low-resource constraints remains an open challenge, as standard fine-tuning is computationally expensive and prone to overfitting on high-dimensional speech data. We propose Convex Language Detection (CLD), a novel framework that integrates theoretically grounded convex optimization techniques into the spoken dialogue systems pipeline. Our method is efficiently implemented via multi-GPU Alternating Direction Method of Multipliers (ADMM) in JAX, thus providing global optimality guarantees and fast training in polynomial time. Theoretically, we prove that our convex objective induces certified margin stability and provide guarantees against feature perturbations. Empirically, we demonstrate sample efficiency and robustness to input dialectical variation, achieving 97-98% accuracy in challenging low-resource regimes. Our open-source package is available at https://pypi.org/project/jaxcld/

1. Introduction

Spoken dialogue systems often misidentify languages in diverse accented and dialectal speech, while scarce and costly speech data constrain improvement. CLD addresses this low-resource challenge with a convex, sample-efficient detection framework that provides robustness guarantees and strong empirical performance.

  • Motivation: Current ASR systems frequently misidentify input languages when speech contains diverse accents and dialectal variation.These errors can propagate into downstream dialogue processing.
  • Motivation: Fine-grained dialects remain under-represented because speech datasets rarely annotate detailed human speech intonations.Audio collection and curation are also more expensive than text, constraining available training data.
  • Contribution: CLD is a fast, sample-efficient algorithm for robust spoken language classification in low-resource regimes.The framework targets dialect identification within ASR models.
  • Contribution: CLD recasts the detection network as a convex program and proves certified margin stability against hidden-feature perturbations.The resulting certificate is computable, data-dependent, and guarantees label invariance within a specified radius.
  • Results: 97-98% accuracy is achieved in low-resource regimes for Whisper Large v3 and MMS-1B experiments across five languages and twenty-four sub-dialects.CLD remains performant with training datasets containing fewer than one hundred samples and consistently outperforms competitors.
  • Resources: The method is implemented as a pip-installable JAX package with open-source code for reproducibility and deployment.The package is intended to support robust speech-driven tools and continued research.

2. Related Work

Related work spans multilingual ASR, low-resource dialect handling, robustness certification, and convex neural-network reformulations. Existing approaches often depend on substantial fine-tuning data, while convex methods have largely been studied theoretically or on small image benchmarks.

  • Multilingual Tasks: Multilingual ASR models cover many languages but generally perform best on English, with lower-resource performance reduced by limited training data.This motivates research on improving low-resource ASR performance.
  • Low-resource Environments: Dialect variation remains difficult even within high-resource languages such as English and Mandarin.Prior work explores prosody-assisted systems and bidirectional recurrent models for this challenge.
  • Low-resource Environments: Existing low-resource methods are heavily dependent on large fine-tuning datasets and typically use learning rates ten times smaller than standard supervised fine-tuning.This dependence creates a resource-intensive training setting.
  • Certified Robustness and Lipschitz analysis: Prior robustness-certification work commonly analyzes local Lipschitz behavior in already-trained non-convex networks.Related perturbation analyses have also been applied in speech processing.
  • Convex Programs: Convex reformulations of two-layer neural networks provide polynomial-time convergence to global optima but have mainly targeted theory or small-scale image benchmarks.This leaves large-scale speech applications comparatively underexplored.

3. Methodology

CLD converts encoder speech representations into language predictions using a convex detection head trained with scalable optimization. The online module is inserted before decoding to provide fast language-conditioned transcription.

  • Convex Language Detection: CLD formally integrates a convex reformulated two-layer ReLU program with an ASR model architecture.Section 3.1 develops the convex program and Section 3.2 integrates it into ASR.
  • Convex Two-Layer ReLU Networks: The standard two-layer ReLU network uses hidden units with ReLU activations and is trained through a non-convex objective sensitive to hyperparameter tuning.These sensitivities become more consequential for expensive, high-dimensional speech applications.
  • Convex Two-Layer ReLU Networks: The convex reformulation characterizes all ReLU activation patterns induced by the data and replaces the original optimization with a convex program.Exact equivalence requires all patterns; practical training samples patterns, with reported discrepancies negligible in practice.
  • Convex Language Detection Algorithm: CLD extracts hidden representations from the ASR encoder and solves the resulting detection problem with a multi-GPU, batched CRONOS implementation in JAX.This produces a lightweight head operating directly on encoder features to identify the input language before decoding.
  • Low Latency Inference: During inference, the encoder representation is pooled, classified by the trained convex head, and supplied as a language token to the decoder.The prediction uses a single lightweight forward pass designed to maintain sub-500ms latency.

4. Theoretical Analysis

The theoretical analysis recasts CLD as a convex-program-trained detection head and establishes Lipschitz and margin-stability guarantees for hidden-feature perturbations. The resulting certificate can be computed from the learned model and interpreted primarily in feature space.

  • 4.1. Margin Stability in Hidden Features: CLD trains a lightweight detection head on ASR encoder features using a convex program.The head is a multi-class classifier f: R^d → R^K, with a finite two-layer ReLU representation under the cvxNN construction.
  • 4.1. Margin Stability in Hidden Features: A perturbation δ satisfying ∥δ∥2 < mar(h, y)/(2∥f∥var) leaves the predicted class unchanged.The guarantee applies to hidden features h and depends on the initial one-vs-rest margin and the variation norm.
  • 4.2. Robustness Certificates from the Convex Program: With an LE-Lipschitz encoder, audio perturbations preserve the predicted class within radius mar(E(x), y)/(2∥f∥varLE).For deep Transformer encoders, global LE bounds may be highly pessimistic, so the paper treats audio-space certificates as conservative diagnostics.
  • 4.1. Margin Stability in Hidden Features: The variation norm controls both the logit Lipschitz constant and classification-margin stability under hidden-feature perturbations.This links the convex objective to a quantitative robustness analysis of the detection head.
  • 4.2. Robustness Certificates from the Convex Program: The convex penalty provides a data-dependent upper bound on ∥f∥var, allowing the certified Lipschitz constant and margin radius to be read from learned weights.The bound can be expressed using block or Frobenius norms; larger β tightens the certified radius under the stated penalty choice.
  • 4.2. Robustness Certificates from the Convex Program: The learned convex solution admits an equivalent vector-valued two-layer ReLU representation with at most 2PK hidden units.This representation connects the convex formulation to the variation-norm certificate.

5. Experiments

Experiments evaluate CLD across ASR models, languages, dialects, sample sizes, and qualitative dialogue settings. Across these tests, CLD shows strong low-resource accuracy, robustness to dialect variation, reduced decoding errors, and substantially lower training cost.

  • Experimental setup: CLD is evaluated against default language detection, neural-network baselines, and multiple classifiers across Whisper-Small, Whisper-Large-V3, and MMS-1B.The evaluation reports WER, CER, detection accuracy, training time, and computational efficiency across five languages.
  • Low-resource performance: 96.94% minimum and 99.14% maximum detection accuracy show CLD remains consistent across training sample sizes.The reported range spans experiments using 100, 500, 1,000, and 10,000 samples.
  • Error analysis: CLD reduces cross-lingual decoding failures by supplying the predicted language token before decoding, while within-language dialect transcription errors remain.For Whisper-Small, WER drops from 139.37 with the default detector to 31.74 with CLD in the multiclass setting.
  • Training efficiency: 64.45 seconds of training time, approximately 7.7% of vanilla-NN runtime and 13x fewer TFLOPs, demonstrate CLD’s computational efficiency.The convex ADMM formulation provides parallelizable updates, rapid convergence, and a unique global optimum without heuristic hyperparameter tuning.
  • Dialect variation: 88.73% accuracy on Min Dong Chinese exceeds the 9.86% default and 25.35% fine-tuned neural-network results.CLD exceeds 94% accuracy across the other evaluated dialects.
  • Multiclass classification: 44.78% higher detection accuracy and 12.74% lower WER than the MMS-1B baseline establish CLD as the best-performing method across reported metrics.The experiments also report scaling potential when CLD augments larger pretrained models.

6. Conclusion

CLD is presented as a fast, lightweight, theoretically grounded method for robust language identification in ASR, with certified robustness and improved sample efficiency. The paper identifies end-to-end differentiation and integration into ASR as future directions.

  • CLD is a fast, lightweight, theoretically grounded method for robust language identification in ASR tasks.
  • Its convex reformulation provides certified robustness and improved sample efficiency across dataset sizes.
  • Future work will make the full pipeline differentiable and integrate CLD more deeply into ASR and multimodal agentic models.

Impact statement

The paper frames CLD as a practical step toward improving access to spoken dialogue models for users with diverse speech variations. Its stated goal is more equitable access across multicultural backgrounds.

  • CLD is positioned as a practical, efficient, and effective method for improving accessibility to spoken dialogue models.
  • The framework targets spoken-dialogue systems whose access is limited by insufficient understanding of input dialect.
  • The authors aim to increase equitable access for a broader society of diverse speech variations.

A. Proof of Main Results and certificates for margin stability

This section connects the convex program in Eq. 2 with a two-layer ReLU representation. That connection produces computable certificates expressed as certified radii.

  • The proof links the convex program in Eq. 2 to a two-layer ReLU representation.
  • The linked representation enables computable certificates for the model.
  • These certificates translate directly into certified radii.

A.1. Proof of Lemma 4.3

The proof develops variation-norm and activation-pattern machinery to bound logit changes under hidden-feature perturbations. It also connects the convex objective and encoder Lipschitzness to computable end-to-end certificates.

  • Activation-pattern representation: The proof uses sampled activation patterns and pattern cones to characterize two-layer ReLU representations.
  • Variation-norm bound: The standard two-layer ReLU variation norm bounds logit changes by ∥f∥var∥h −h′∥2.
  • Convex certificate: Feasible solutions of the convex program yield explicit upper bounds on ∥f∥var and therefore Lipschitz certificates for the logits.
  • End-to-end certificate: When the encoder E is L_E-Lipschitz, the hidden-feature bound extends to an end-to-end certificate using L_E∥x −x′∥2.
  • Certificate control: Increasing β tightens the certificate bound linearly.

C. Alternating Direction Method of Multipliers

ADMM solves CLD’s large convex optimization problem by decomposing it into linked local subproblems, enabling parallel optimization and global-optimum guarantees. In CLD, this supports scalable training for high-dimensional ASR features and reduces computational cost.

  • ADMM decomposes large convex optimization problems into smaller local subproblems linked by Lagrange multipliers and a quadratic penalty term.This structure is especially suited to separable objectives and constraints.
  • ADMM is presented as a significant novelty in applying convex optimization to large-scale spoken dialogue systems.The paper distinguishes this application from ADMM’s established use in control and optimization.
  • CLD uses ADMM to make its convex reformulation of a two-layer ReLU network tractable for high-dimensional ASR hidden features.The implementation supports batched, multi-GPU parallel optimization in JAX.
  • ADMM training is described as converging to a unique global optimum, avoiding local-minimum issues and brittle learning-rate dependence.This contrasts with the standard neural-network training process described for the baseline.
  • 13x fewer TFLOPs than the standard vanilla-NN is reported for CLD, alongside drastically reduced training time.

D. Extended Experimental Results and Discussion

The extended results compare language-classification behavior across models, languages, and training sample sizes. The figures specifically examine CER and classification performance for Whisper-small systems and CLD-related comparisons.

  • Figure D.1 compares CER for Whisper-small default, Whisper-small fine-tuned, vanilla-NN, and CLD on binary classification.
  • The extended comparison spans English and Chinese across training sample sizes of 100, 500, 1000, and 10 000.
  • Figure D.2 compares language-classification performance between WSP-SFT and WSP on Whisper-small across English, Chinese, Indonesian, Malaysian, Hindi, and other predicted languages.

D.1. Classification Accuracy per Sample Size Ablation

The ablation evaluates multi-dialect classification across sample sizes, accents, models, and tuning settings. It also documents the datasets, validation protocol, classifier baselines, and experimental hardware used for comparison.

  • Classification accuracy per sample size: Tables D.1–D.3 report English–Chinese classification accuracy across 10 accents at 100, 1000, and 10 000 samples per language.The corresponding samples per dialect are 20, 200, and 2000.
  • Classification accuracy per sample size: Table D.4 reports vanilla-NN accuracy after hyperparameter tuning across 100, 500, 1000, and 10000 samples, including changes from the non-tuned model.
  • Extended metrics: Table D.5 compares WER, CER, and detection accuracy for WSP, WSP-L, and MMS-1B using vanilla-NN.
  • Experimental setup: The study focuses on dialect datasets and evaluates WER, CER, language-detection accuracy, wall-clock training time, and computational efficiency across Whisper-small, Whisper-Large-V3, and MMS-1B.
  • Additional comparisons: Whisper-small default language-detection accuracy is broken down by language and accent, while a human case study compares Default, vanilla-NN, and CLD.The case study includes five Singapore participants for English and ten southeastern China participants for Mandarin.
Loading 2605.23235v1…