PaperlayerView exact revision on arXiv

Source-linked AI summary

Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages

Kuan-Tang Huang, Cheng-Yeh Yang, Chien-Chun Wang, Hung-Shin Lee, Hsin-Min Wang, Berlin Chen

arXiv:2608.29239v1cs.CLcs.SDeess.AS

TL;DR

Low-resource ASR lacks sufficient target-language transcripts for reliable decoder generation. SAMA-ASR addresses this with a frozen-backbone adapter that combines translation-derived semantic anchors and speech-derived acoustic grounding, and experiments show gains over acoustic, prompt-based, and semantic-only baselines on Taiwanese Hokkien and Hakka, including with automatically generated anchors.

  • Problem

    Scarce target-language transcripts provide limited supervised evidence for lexical choices, orthographic conventions, constructions, and meaning-compatible continuations in low-resource ASR.

  • Method

    SAMA-ASR adds a lightweight decoder-side adapter that lets autoregressive states attend to translation-derived semantic embeddings and a speech embedding while keeping the pretrained ASR backbone frozen.

  • Results

    SAMA-ASR improves over acoustic, prompt-based, and semantic-only translation-guided baselines on Taiwanese Hokkien and Hakka, while compact upstream ST models provide useful automatic anchors.

  • Takeaways & Limitations

    Acoustic-grounded semantic anchoring offers a practical way to exploit paired speech–translation data when target-language transcripts are scarce.

  • Takeaways & Limitations

    The practical pipeline increases inference cost and depends on the quality of automatic semantic anchors; it also assumes paired speech–translation data during adaptation.

Abstract

from arXiv · show

Low-resource ASR remains difficult because scarce transcripts provide limited supervised evidence for target-side generation. To address this gap, we propose SAMA-ASR, a lightweight adapter mechanism that augments the decoder with semantic anchors from auxiliary translations and an acoustic anchor from speech; in principle, the mechanism can be applied to similar encoder--decoder multitask speech models. Through cross-modal adaptation, SAMA-ASR conditions decoder states on translation-derived semantic embeddings and a speech embedding, combining utterance-level meaning with speech-grounded evidence before token prediction. At evaluation time, these semantic anchors can be generated automatically by an upstream speech-to-text translator rather than supplied as oracle translations. Experiments on two 30-hour datasets covering the low-resource Sinitic varieties Taiwanese Hokkien and Hakka show that SAMA-ASR improves over acoustic, prior prompt-based, and semantic-only translation-guided baselines and remains effective in practical automatic semantic-anchor settings; translator-capacity analyses show that useful semantic anchors can be produced by a compact ST model.

1 Introduction

SAMA-ASR addresses scarce target-language transcripts by adding translation-derived semantic anchors and speech-derived acoustic grounding to a frozen ASR decoder. Experiments on Taiwanese Hokkien and Hakka support automatic semantic-anchor generation with compact upstream translators.

  • Method: The decoder-side adapter conditions autoregressive states on translation-derived semantic embeddings and an acoustic speech anchor.The acoustic path helps constrain semantically plausible but acoustically unsupported continuations.
  • Method: Semantic anchors provide global meaning context before partial autoregressive histories become reliable.The design is intended to reduce exposure to early error accumulation during decoding.
  • Motivation: SAMA-ASR reframes extremely low-resource ASR as target-side generation assisted by paired speech–translation data.This supplies semantic supervision when target-language transcripts are scarce.
  • Inference: Automatic upstream speech-to-text translation removes the need for oracle translations at test time.A compact Whisper Small ST model provides useful anchors with moderate added model cost relative to the 769M-parameter Whisper Medium ASR backbone.
  • Evaluation: The framework is validated on two low-resource Sinitic varieties, Taiwanese Hokkien and Hakka, across data scales and translator capacities.The analyses also examine auxiliary languages and multilingual anchor compositions.

2 Related Work

Prior work mitigates low-resource ASR through augmentation, resource construction, language-specific modeling, contextual biasing, distillation, and multimodal or text-guided adaptation. SAMA-ASR differs by using runtime translation anchors with explicit acoustic grounding and automatic anchor generation.

  • Resource scarcity: Acoustic augmentation and loosely aligned resource construction reduce annotation costs but can leave segmentation or label noise.Examples include speed perturbation, SpecAugment, subtitles, and audiobook-derived resources.
  • Language-specific adaptation: Language-specific ASR methods exploit phonology, scripts, orthography, or related-language resources, but their assumptions may not transfer unchanged.SAMA-ASR is presented as less tied to handcrafted linguistic features or a single language setting.
  • Adapter methods: Joint ASR–ST and parameter-efficient methods exchange cross-task information or adapt models efficiently, whereas SAMA-ASR keeps the backbone frozen and injects anchors.Its semantic and acoustic evidence is added through a decoder-side adapter rather than joint ASR–ST optimization.
  • Contextual and translation guidance: Contextual biasing typically requires phrase lists, target-language text, or task-specific inventories, limiting suitability for open-domain low-resource ASR.SAMA-ASR instead uses utterance-level translation-derived semantic anchors.
  • Distillation and text fusion: Unlike teacher-guided distillation, SAMA-ASR uses translated text as a runtime semantic input during recognition.It also differs from source-text-assisted ASR because its practical pipeline first generates auxiliary translations from speech.

3 Methodology

SAMA-ASR is a lightweight decoder-side adapter that addresses scarce target-language transcripts by conditioning autoregressive ASR on translation-derived semantic anchors and a speech-derived acoustic anchor. It uses frozen pretrained components, automatically generated anchors for practical inference, and gated cross-attention before decoder self-attention.

  • Problem framing: SAMA-ASR reframes low-resource ASR as target-side generation under limited transcript supervision and augments decoder conditioning with semantic and acoustic evidence.Scarce transcripts can bias decoding toward locally plausible tokens that conflict with utterance meaning or speech.
  • Anchor construction: The anchor set combines semantic embeddings from auxiliary translations with a speech embedding from the frozen speech encoder.The main setting uses one Mandarin auxiliary translation, while the formulation supports multiple translations.
  • Semantic anchor generation: During practical inference, an independently fine-tuned speech-to-text translation model generates pseudo-translations that supply semantic anchors before ASR decoding.Training uses paired oracle translations, whereas each test utterance is processed by the upstream ST generator.
  • SAMA-ASR architecture: SAMA-ASR inserts parallel, independently gated cross-attention branches into each decoder block so hidden states can query translation and speech anchors.The inserted module is placed before decoder self-attention, allowing anchor information to enter before the frozen decoder updates its token-history representation.
  • SAMA-ASR architecture: The gated anchor contexts and a gated feed-forward branch modify the decoder state, while zero-initialized gates make each inserted module initially behave as an identity adapter.Only SAMA-ASR modules and projection layers are optimized, preserving the pretrained backbone and reducing overfitting risk under scarce transcripts.

4 Experimental Setup

The experiments evaluate SAMA-ASR on two controlled low-resource Sinitic ASR settings with paired dialect speech and Mandarin auxiliary translations. Comparisons include acoustic-only adaptation, practical automatic-translation baselines, semantic-only guidance, and LoRA composition.

  • Datasets: SAMA-ASR is evaluated on Taiwanese Hokkien and Hakka, using paired dialect speech and Mandarin auxiliary translations in both settings.The Hakka data form a controlled 30-hour subset, matched to the scale of the Taiwanese Hokkien corpus.
  • Datasets: YT-THDC pairs Taiwanese Hokkien YouTube drama audio with open-caption Mandarin translations used as loosely aligned auxiliary translations.These translations are treated as auxiliary information rather than deterministic pseudo-transcripts.
  • Comparisons: Table 1 compares audio-only adaptation, prior text-guided baselines under automatic Mandarin, and composition with LoRA.The prompt baseline uses Whisper’s native prompt interface, TG-ASR provides semantic-only guidance, and SAMA-ASR adds an acoustic anchor.

5 Results and Discussion

Across practical and controlled evaluations, SAMA-ASR improves low-resource ASR through complementary semantic and acoustic anchoring, while its robustness depends on anchor fidelity and sufficient paired supervision.

  • Main Results: 23.48% and 22.48% CER show SAMA-ASR outperforming the semantic-only TG-ASR baseline on YT-THDC and HAT in the practical automatic-translation setting.TG-ASR reaches 24.38% and 24.97% CER, respectively; SAMA-ASR adds explicit acoustic grounding.
  • Autoregressive Cold-Start Analysis: The first output positions are hardest, while SAMA-ASR's advantage over TG-ASR emerges from positions 2–3 onward rather than at the first position.This pattern is observed across both datasets and supports a role for semantic anchors during fragile early autoregressive conditioning.
  • Robustness: On HAT, Small has higher translator BLEU than Medium (45.90 vs. 43.41), but lower mBERT cosine similarity (0.838 vs. 0.864) and worse CER (24.05 vs. 22.48).Base and larger translators outperform the context-only baseline on both datasets, so BLEU is not a monotonic proxy for downstream utility.
  • Robustness: With 1 hour and 10 hours of paired data, SAMA-ASR gives the best CERs on both datasets, but Tiny translators or 10-minute data can cross its practical boundary.The method requires sufficient paired supervision or meaning-compatible upstream translations to offset uncertainty in low-resource decoding.
  • Semantic Prior Ablation: Replacing frozen mBERT with random or Whisper embeddings substantially degrades audio-enhanced SAMA-ASR, indicating dependence on cross-lingual semantic structure.The text encoder constrains decoder hypotheses while acoustic evidence determines the final surface form.
  • Robustness: All MT-derived auxiliary languages, including English, Spanish, and French, outperform the context-only baseline, but non-Sinitic gains do not follow a simple monotonic relation with typological distance.The auxiliary-language experiment uses translations derived from oracle Mandarin translations, making it a controlled diagnostic rather than a deployment-time claim.

6 Conclusion

SAMA-ASR frames low-resource ASR as both acoustic adaptation and target-side generation, using translation-derived semantic anchors while retaining an acoustic anchor. Across Taiwanese Hokkien and Hakka, it improves over acoustic, prompt-based, and semantic-only baselines, while compact upstream ST models provide useful automatic anchors.

  • SAMA-ASR treats low-resource ASR as both acoustic adaptation and target-side generation.
  • The decoder conditions a frozen ASR backbone on translation-derived semantic anchors and an explicit acoustic anchor.
  • Across Taiwanese Hokkien and Hakka, SAMA-ASR consistently improves over acoustic, prompt-based, and semantic-only TG-ASR baselines.
  • Compact upstream ST models can generate useful semantic anchors without oracle translations at test time.
  • Combining SAMA-ASR with LoRA yields further improvements.

Limitations

SAMA-ASR increases inference cost and depends on semantic anchors remaining sufficiently meaning-compatible. The framework is outside scope when paired auxiliary translations are unavailable.

  • Practical inference requires an upstream ST pass, semantic encoding, and SAMA-ASR decoding, increasing cost over context-only or audio-only baselines.The offline real-time factor remains below 1.0, but the authors identify latency–accuracy optimization as future work.
  • Noisy or hallucinated semantic anchors can reduce SAMA-ASR’s benefit or mislead the decoder.Useful anchors need not be oracle translations, but they must remain sufficiently meaning-compatible.
  • Settings without paired auxiliary translations require another source of semantic supervision and remain outside the framework’s scope.

A Additional Experimental Details

The supplementary details document artifact availability, dataset statistics, decoding settings, computational resources, and AI-assistance disclosure. Experiments use publicly available models and datasets under their intended research purposes and academic licenses.

  • All pretrained models and datasets used in the experiments are publicly available and used consistently with their intended research purposes and licenses.
  • The benchmark datasets were curated to exclude personally identifiable information and offensive content.
  • Table 5 reports low-resource dataset statistics with durations measured in hours.
  • Prompted utterances are decoded one at a time with the prompt reset for each utterance.The decoding configuration uses no timestamps, beam size 1, and temperature 0.
  • The complete architecture has approximately 1.3B parameters, with roughly 404M trainable during training.A single RTX 3090 is used, and one experimental run requires approximately 12 GPU hours.
  • Prism AI and Gemini assisted with language polishing, grammar correction, and code generation, with outputs reviewed and tested by the authors.

B Robustness under Limited Training Data and Noisy Translation Resources

Robustness analyses separate train–test translation-source mismatch from translation noise. With sufficient paired data, the mismatch is small, while stronger automatic translators progressively improve performance and automatic translations remain near oracle semantic neighborhoods.

  • Limited Training Data: Under the full 30-hour setting, oracle- and pseudo-translation training are close and mixed across YT-THDC and HAT.Oracle training is slightly better on YT-THDC, while pseudo-translation training is better on HAT.
  • Limited Training Data: With reduced paired data, oracle-translation training consistently outperforms pseudo-translation training because generated translations become noisier.
  • Limited Training Data: The main experiments use oracle-translation training to avoid propagating translator errors, while practical evaluation uses automatic translations.
  • Noisy Translation Resources: SAMA-ASR remains effective when pseudo translations are used during both training and evaluation.Even Tiny improves slightly over the Cross Attn. baseline, while stronger ST models progressively reduce CER as translation quality improves.
  • Noisy Translation Resources: Reducing paired data weakens both adaptation and translation, whereas the noisy-resource setting isolates translation noise with the full training set retained.
  • Noisy Translation Resources: Automatic translations for the same utterance tend to remain close to oracle translations in mBERT UMAP neighborhoods.The visualization uses five sampled test utterances per dataset; colors denote utterances and marker shapes denote translation sources.

D Detailed Case Study

The case study shows that SAMA-ASR benefits from combining semantic guidance with acoustic grounding, while auxiliary-language gains depend on translation composition rather than language count alone.

  • D Detailed Case Study: Self Attn. and Cross Attn. preserve local phonetic similarity but produce semantically incompatible substitutions for 「到 銀行」.Self Attn. predicts 「教囡仔」, while Cross Attn. predicts 「教議員學」.
  • D Detailed Case Study: TG-ASR uses semantic context but predicts the acoustically unsupported phrase 「交 警方」 after the nearby mention of “government agencies.”This illustrates the failure mode of semantic guidance without an explicit acoustic anchor.
  • D Detailed Case Study: SAMA-ASR recovers 「到銀行」 by combining semantic guidance with acoustic grounding, unlike baselines that make acoustically or semantically plausible substitutions.The case study contrasts pronunciation-driven errors from Self Attn. and Cross Attn. with text-guided semantic hallucination from TG-ASR.
  • Auxiliary-Language Analysis: Mandarin has the lowest syntactic and phonological distances to Taiwanese Hokkien and Hakka, consistent with strong oracle-translation performance.The broader comparison does not establish a simple monotonic relationship between typological distance and gains.
  • Multilingual Anchor Composition: Best-first multilingual anchoring consistently outperforms worst-first, especially at L = 2, while adding all five languages degrades performance.The results indicate that high-quality complementary translations help, whereas redundant or less compatible translations can dilute the attention signal.

F Inference Efficiency and Deployment Cost Analysis

SAMA-ASR adds measurable inference cost but remains faster than real time, with automatic-anchor deployment offering a trade-off between translator capacity and efficiency.

  • Measurement Protocol: Efficiency measurements use batch size 1, exclude the first 10 warm-up utterances, and apply greedy decoding per utterance.RTF is wall-clock processing time divided by input-audio duration.
  • Automatic-Anchor Deployment: End-to-end deployment generates an automatic Mandarin semantic anchor with an upstream ST model before mBERT encoding and SAMA-ASR decoding.The reported deployment table excludes MT-based multilingual-anchor expansion.
  • Provided-Translation Cost: 16.7% relative CER reduction over TG-ASR accompanies 8.1% more parameters, 7.4% more peak GPU memory, and RTF rising from 0.307 to 0.391.These measurements isolate anchoring-module overhead when translations are already available.
  • Automatic-Anchor Deployment: All evaluated automatic-anchor configurations remain faster than real time while larger ST models improve downstream CER at higher latency, model size, and memory cost.The Small translator provides a lower-cost operating point at 0.489 RTF; the Medium translator reaches 23.48% CER at 0.603 RTF.
Loading 2608.29239v1…