PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Data Sampling Framework for Multilingual ASR via Linguistic-Aware Error-Driven Optimization

Zhiheng Qian, Junhao Du, Jing Peng, Hanqi Li, Kunyang Peng, Feng Xue, Lu Chen, Shuai Wang, Kai Yu

arXiv:2610.36538v1cs.CL

TL;DR

Multilingual ASR sampling methods often use predefined criteria and operate at one granularity, while linguistic error diagnoses are not routinely converted into training-data decisions. This paper puts an LLM agent in the training loop to use linguistic reports and training state for hierarchical sampling. Across twelve Common Voice languages, the method reports lower average error rates than no sampling and random sampling, with its best result on nine languages.

  • Problem

    Existing sampling approaches generally use predefined criteria at one granularity, while WER/CER-based evaluation does not identify linguistic error sources for adaptive data selection.

    Source: §2.1 · p. 3, §2.2 · p. 4

  • Method

    An LLM agent uses phonological, morphological, and syntactic error diagnostics and training information to select domain-, sample-, and token-level sampling strategies.

    Source: §1 · p. 2, §6 · p. 13

  • Results

    The agentic-300h setting achieves an average error rate of 19.4, compared with 23.9 for no sampling and 22.5 for random sampling, and obtains the best result on nine languages.

    Source: §6 · p. 13

  • Takeaways & Limitations

    The results suggest that multilingual ASR evaluation can provide actionable feedback for subsequent data selection, as well as measure performance.

    Source: §6 · p. 13

    Word-order deviations were too few to support a reliable conclusion about global syntactic modeling.

    Source: §5.2 · p. 13

Abstract

from arXiv · show

Multilingual automatic speech recognition (ASR) has achieved rapid progress with large-scale pretrained models and multilingual speech corpora. Previous studies have proved the pivotal role of data sampling in multilingual ASR performance. However, existing sampling strategies largely rely on time-consuming manual, heuristic-based designs. To address this, we propose a linguistic-aware agentic data sampling framework that introduces a data sampling LLM agent into the multilingual ASR training loop. The agent is driven by a linguistically grounded rubric that analyzes ASR errors from phonological, morphological, and syntactic perspectives, and uses the resulting feedback together with training statistics, budget constraints, and algorithm profiles to select domain-level, sample-level, and token-level sampling strategies. This design turns evaluation results into adaptive data-selection decisions rather than treating WER/CER as final endpoints only. We conduct experiments on twelve Common Voice languages using Whisper-medium as the ASR backbone. The results show that our method achieves performance better than full-data training while using less training data and lower computational resources, and substantially accelerates training convergence compared to conventional sampling strategies.

1 Introduction

Multilingual ASR data differ in availability and linguistic and acoustic properties, while existing sampling methods rarely adapt language allocation, utterance selection, and token supervision together to evolving errors. The proposed framework uses linguistic diagnostics to guide sampling decisions and reports stronger average performance than no-sampling and random-sampling baselines under the same cumulative training budget.

1 Introduction

  • Multilingual corpora vary in data availability, acoustic conditions, pronunciation, morphology, and writing conventions, contributing to uneven recognition performance across languages.Prior approaches typically optimize a predefined criterion at one granularity rather than jointly adapting language, utterance, and token choices.Source: §1 · p. 2
  • The framework analyzes phonological, morphological, and syntactic errors alongside per-language performance, training history, budget constraints, and algorithm profiles to choose domain-, sample-, and token-level strategies.Sampling becomes an adaptive feedback loop rather than fixed preprocessing.Source: §1 · p. 2
  • The method outperforms no-sampling and random-sampling baselines in average multilingual recognition under the same cumulative training budget, with the best results on most evaluated languages.Experiments cover twelve Common Voice languages.Source: §1 · p. 2
  • The framework improves training convergence and reduces phonological, morphological, and function-word errors as the model’s error profile evolves.These analyses also report adaptive changes in the agent’s decisions.Source: §1 · p. 3

2 Related Works

Prior work broadens multilingual evaluation and data selection across languages and sampling granularities, but linguistic error diagnoses are generally used for benchmarking or post-hoc interpretation rather than to adapt training data. The paper positions its approach as closing that gap with structured error feedback for subsequent sampling decisions.

2.1 Evaluation and Error Diagnosis in Multilingual ASR

  • Multilingual benchmarks have expanded from separate language evaluations to broader cross-lingual, multi-dataset, and varied-task comparisons.Common Voice provides diverse community-contributed speech; FLEURS offers parallel content across languages.Source: §2.1 · p. 3
  • WER and CER support reproducible transcription comparisons, but their interpretation varies with language and evaluation design, and neither identifies the linguistic source of errors.Phonological, morphological, and syntactic analyses can expose differences that aggregate edit distance alone misses.Source: §2.1 · p. 3
  • Evaluation and diagnostic studies mainly benchmark or interpret errors after training, whereas this work converts phonological, morphological, and syntactic statistics into feedback for domain-, sample-, and token-level selection.The related work identifies coordination across these granularities according to changing ASR errors as underexplored.Source: §2.1 · p. 3, §2.2 · p. 4

2.2 Data Sampling Algorithms

  • Existing multilingual sampling methods use domain reweighting, example selection, token-level control, and speech-specific linguistic or acoustic similarity signals.These approaches show that data quantity or loss alone does not fully determine the usefulness of multilingual speech.Source: §2.2 · p. 4

3 Method

The method places an LLM agent in an iterative ASR training loop, using linguistic error reports and training state to select hierarchical data-sampling actions. Its rubric diagnoses phonological, morphological, and syntactic errors, while a profiled algorithm pool supports decisions at domain, sample, and token levels.

3 Method

  • At each epoch, the system fine-tunes the ASR model, evaluates it on development data, converts recognition outputs into a linguistic report, and uses that report and training statistics to plan the next round’s sampling.The plan spans domain, sample, and token decisions.Source: §3 · p. 5

3.1 Linguistic Rubric

  • The rubric classifies errors by phonology, morphology, and syntax to explain observed errors and guide targeted sampling rather than replace WER or CER.It analyzes phonemes, inflectional or other morphological markers, and prepositions, function words, and local word order; reports combine categorical distributions and short summaries with WER/CER and training statistics.Source: §3.1 · p. 6

3.2 Data Sampling Agent

  • The sampling agent selects compatible algorithms and parameters at domain, sample, and token levels, corresponding to language ratios, utterance selection, and token operations during optimization.The predefined pool contains seven algorithms and is extensible through modules with declared inputs, behavior profiles, and parameter schemas.Source: Hierarchical Algorithm Pool · pp. 6–7
  • The agent applies its three sampling actions sequentially: adjusting language-level weights, selecting utterances under those weights, then applying token operations.Each action pairs an algorithm choice with its parameters.Source: Data Sampling Procedure · p. 7
  • At each stage, the agent receives epoch statistics, linguistic reports, algorithm profiles, prior-stage execution results, and decision history, then outputs a structured choice with parameters and rationale.Table 1 presents the multi-level data-sampling algorithm pool.Source: Data Sampling Procedure · pp. 7–8, Table 1 · p. 7

4 Experiments

With a fixed 1,200-hour training budget, agentic sampling achieves the best reported average performance and continues reducing overall WER after baseline convergence. Case analyses show corrections across morphology, syntax, and phonology.

4.1 Setup

  • The experiments use a 12-language Common Voice subset and fix total trained data at 1,200 hours across strategies.The corpus has 600 training hours, with 50 hours per language; agentic batches are 100 or 300 hours.Source: Baselines and Implementation · p. 8, Dataset · p. 8

4.2 Main Results

  • 19.4 average error rate for agentic-300h, versus 23.9 for no sampling and 22.5 for random sampling.It yields the lowest error rates in 9 of 12 languages; Spanish and Portuguese WER fall 38.5% and 38.0% versus no sampling.Source: §4.2 · pp. 8–9
  • After about 1,200 training hours, baseline WER converges while agentic sampling continues to reduce overall WER.The comparison uses 300-hour batches for random and agentic sampling.Source: §4.2 · pp. 9–10, Figure 2 · p. 9

4.3 Case Analysis

  • Representative cases show agentic sampling correcting persistent baseline errors in inflection, function-word syntax, and phonology.Examples include restoring French adjective plural agreement and an Italian prepositional construction; a phone-error diagnosis also guides sampling toward a persistent phonological confusion.Source: §4.3 · pp. 10–11

5 Analysis

The agent changes its sampling decisions as linguistic diagnostics and evaluation history accumulate, selecting actions at different data granularities. Linguistic error counts show reductions across error types, with the size and reliability of gains varying by linguistic level.

5.1 Agent Decision Behavior

  • After a broad cold-start policy, the agent changes its choices in response to evolving error diagnostics rather than repeating a fixed schedule.Its decisions connect phonological errors to sample selection and distinguish domain allocation from sample- and token-level supervision choices.Source: §5.1 · pp. 11–12
  • The observed decisions are traceable and responsive to model feedback, but do not establish that every selected strategy is globally optimal.The agent revises both the target and granularity of data selection based on linguistic diagnostics.Source: §5.1 · p. 12

5.2 Linguistic Error Reduction

  • Agentic sampling reduces errors across linguistic levels; phonological errors have the largest absolute reductions, while morphological errors show the largest relative improvement.Preposition and function-word errors also decline, but the number of word-order deviations is too small for a reliable conclusion about global syntactic modeling.Source: §5.2 · pp. 12–13, Table 5 · p. 12

6 Conclusion

The framework turns phonological, morphological, and syntactic error diagnostics into adaptive sampling decisions at domain, utterance, and token levels. On twelve Common Voice languages with Whisper-medium, agentic-300h outperforms both baselines on average and is best on nine languages.

6 Conclusion

  • The framework converts phonological, morphological, and syntactic diagnostics into domain-, utterance-, and token-level sampling decisions.Evaluation is placed inside the training loop, making sampling adaptive rather than a fixed preprocessing strategy.Source: §6 · p. 13
  • 19.4 average error rate for agentic-300h, compared with 23.9 for no sampling and 22.5 for random sampling; it is best on nine languages.Experiments use twelve Common Voice languages with Whisper-medium.Source: §6 · p. 13
  • The reported results suggest multilingual ASR evaluation can guide subsequent data selection, not only measure performance.The selected strategies change with the evolving model error profile, alongside reductions in phone-level, inflectional, and function-word errors.Source: §6 · p. 13

Appendix 1

The appendix illustrates the structure of agent decisions at each sampling stage with an example language-level choice.

Appendix 1

  • The decision-structure example identifies its scope as the agent's choices at each stage.The passage labels the material as the structure of agent decision in each stage.Source: Appendix 1 · p. 17
  • The example decision cites high variation in per-language CER and smooths low-resource-language weights with alpha=0.8.It names DoReMi and gives reweight_eta=2.0 among its parameters.Source: Appendix 1 · p. 17
  • The example includes a DoReMi algorithm selection with alpha=0.8 and reweight_eta=2.0.The snippet also mentions smoothing weights for low-resource languages.Source: Appendix 1 · p. 17

Appendix 2

The linguistic-rubric example organizes error summaries across phonology, morphology, and syntax, with entries spanning phones, inflectional markers, and function words.

Appendix 2

  • The example's phonology rubric records phone error rates and associated languages, with further biphone and triphone error entries.One listed phone is “i,” with an error rate of 0.68 across be, pl, and pt.Source: Appendix 2 · p. 17
  • The rubric example is organized into phonology, morphology, and syntax categories.Source: Appendix 2 · p. 17
  • The morphology example reports an inflectional marker “er” for German with an error rate of 0.10.Its summary describes inflectional errors as concentrated on particular patterns.Source: Appendix 2 · p. 17
  • The syntax example reports French preposition “de” with an error rate of 0.13 and notes frequent preposition and function-word errors.Source: Appendix 2 · p. 17

Appendix 3: Prompt Template

The prompt configures an LLM agent for sequential sampling stages, supplying diagnostic and run context and imposing algorithm-selection and parameter rules.

Appendix 3: Prompt Template

  • The template identifies the agent as a multilingual ASR data-sampling agent and configures it for a named stage, its available algorithms, and its phase.Source: Appendix 3: Prompt Template · p. 18
  • The workflow runs three stages sequentially, with each stage's output feeding into the next.Source: Appendix 3: Prompt Template · p. 18
  • Stage inputs include rubric diagnoses, prior-stage summaries, and experiment constraints with algorithm-usage history.The rubric input includes error patterns, per-language and overall CER, phone or grapheme distributions, and summary text.Source: Appendix 3: Prompt Template · p. 18
  • The prompt asks the agent to choose one available algorithm, set schema-specific parameters, and favor conservative settings unless diagnosis supports aggressive filtering.Source: Appendix 3: Prompt Template · p. 18
  • The prompt directs exploration across algorithms and stage combinations, discouraging repeated use of one stage algorithm for more than two consecutive epochs unless alternatives are invalid.Source: Appendix 3: Prompt Template · p. 18
  • Model-based score fields are precomputed from the current epoch checkpoint and fixed train split when missing, while specified file paths must refer to real run artifacts.Source: Appendix 3: Prompt Template · p. 18

Related papers

Loading 2610.36538v1…