Benchmarking Automatic Speech Recognition Tools for Iberian Languages

1. Introduction

• Iberian languages are unevenly represented in ASR: Spanish and Portuguese are comparatively well-resourced, Catalan intermediate, and Basque and Galician under-resourced.Source: §1 · p. 1
• Existing ASR benchmarks either compare many models across few languages or cover many Iberian languages through text understanding rather than speech.Albayzin covers Spanish broadcast and Basque-Spanish code-switching, while IberoBench is a text benchmark.Source: §1 · p. 1
• Performance depends on conditions, so no single model is best everywhere, and a single corpus cannot support robust conclusions.Source: §1 · p. 1
• Aggregate WER can mask demographic disparities; reporting per-language and per-sex performance is presented as essential to a fair, complete assessment.Source: §1 · p. 1
• The benchmark evaluates ten open-weight systems and one commercial API across five Iberian languages plus German and Turkish, reporting WER, RTF, and RTFx.It also analyzes accuracy-efficiency trade-offs and per-language and per-sex performance.Source: §1 · p. 1

2. ASR benchmark

• The benchmark was assembled by selecting language datasets, choosing open-weight models with broad or subset language support, and generating hypotheses and measuring latency for each model.Source: §2 · p. 1

2.1. Datasets

• Datasets span read speech, broadcast media, and audiobooks, but domain and recording-condition differences confound cross-language comparisons with intrinsic language difficulty.The design cannot separate these effects; Turkish and German provide controls with differing resource availability.Source: §2.1 · pp. 1–2

2.2. Data statistics

• The merged benchmark contains 85.37 hours of audio, with samples generally under 20 seconds and durations centered around 6 seconds, plus a smaller MediaSpeech cluster near 15 seconds.Source: §2.2 · p. 2
• Among 34,036 samples with relevant metadata, 11,128 (33%) are labeled female, 8,144 (24%) male, and 14,764 (43%) unknown.Unknown denotes samples without a sex label; MediaSpeech's Spanish and Turkish subsets are excluded from this analysis.Source: §2.2 · p. 2

2.3. Models

• The evaluation includes ten open-weight models and one commercial API baseline; open-weight models have fewer than 7B parameters, and one model is not commercially usable.The exception is seamless-m4t-v2-large, licensed CC BY-NC 4.0.Source: §2.3 · p. 2

2.4. Evaluation

• Macro-averaged WER is the unweighted mean of per-language scores, giving languages equal contribution despite dataset-size imbalance.References and hypotheses are normalized by lowercasing, removing punctuation, and converting digits to words except for Turkish.Source: §2.4 · p. 2
• Efficiency is reported as RTF and RTFx; RTF below 1.0 or RTFx above 1.0 indicates faster-than-real-time processing.Higher RTFx means greater throughput.Source: §2.4 · p. 2

3.1. Error Composition and Architectural Patterns

• On the shared Omni-lingual encoder, replacing the CTC head with an LLM decoder lowers substitutions (12.37→8.52), deletions (7.63→6.78), and insertions (1.46→1.34).Substitutions dominate the error budget of every functioning system.Source: §3.1 · p. 3

3.2. Speed and WER Trade-offs

• RTFx 103.7 with 21.5% WER makes omniASR CTC 1B v2 the best accuracy–efficiency trade-off among freely usable models; parakeet is faster (151.7) but has 51.9% WER.Voxtral-Mini-4B-Realtime-2602 has the lowest RTFx (1.4) and a 48.5% macro WER; its causal encoder likely incurs overhead in offline batch evaluation.Source: Figure 2 · p. 3, §3.2 · p. 3

3.3. Per-Language and Per-Sex Analysis

• For German, Spanish, and Portuguese, most systems cluster in a narrow WER band, but domain differences make comparisons—including higher Spanish than German WER—caution-worthy.The datasets differ in domain, limiting how confidently these cross-language rankings can be interpreted.Source: §3.3 · p. 3
• Basque is consistently most challenging, and systems without explicit Basque coverage degrade to near-random output; Catalan is also difficult, while Galician and Turkish are intermediate.The omniASR family is most consistent cross-lingually, including on Basque; the gap between Scribe v2 and seamless-m4t-v2-large and open-weight systems is widest on lower-resource languages.Source: §3.3 · p. 3
• Most models show a male WER advantage, largest for canary-1b-v2 (gap +17.1); seamless-m4tv2-large slightly favors female speakers.Scribe v2 has minimal observed sex differences, but opaque training data and dataset imbalances make that parity hard to attribute.Source: §3.3 · p. 3

3. Results

• 6.45% WER for Scribe v2 is the lowest; seamless-m4t-v2-large leads open-weight systems at 8.12%, while omniASR-LLM-1B-v2 leads freely usable models at 16.64%.Table 3 reports macro-averaged WER across languages; seamless-m4t-v2-large is restricted to non-commercial use.Source: §3 · p. 3, Table 3 · p. 4

3.3. Per-Language and Per-Sex Analysis

• The unknown-speaker-sex group contains German only; lower WER there reflects dataset composition rather than intrinsic robustness.MediaSpeech lacks sex labels, so its Spanish and Turkish subsets are excluded from this analysis.Source: §3.3 · p. 4

• Cross-language comparisons warrant caution because datasets differ in domain and recording conditions, and broadcast-derived sets likely inflate WER relative to read speech.Source: §4 · p. 4
• Unconfirmed training overlap with benchmark subsets could optimistically bias model WER.Source: §4 · p. 4
• Identical-hardware inference tests without architecture-specific configuration may disadvantage some systems; Scribe v2’s RTF and RTFx instead reflect API latency.Source: §4 · p. 4
• Language labels can conceal dialectal variation, including Brazilian versus European Portuguese.Source: §4 · p. 4
• Controlled demographic-disparity assessment requires speaker-group and language-balanced datasets beyond this benchmark’s scope.Disparities were consistent across architectures, suggesting fairness challenges are pervasive in multilingual ASR.Source: §4 · p. 4

• No single open-weight model dominates across accuracy, speed, language coverage, and licensing.Scribe v2 achieves the best WERmacro, while seamless-m4tv2-large is the strongest open-weight model but restricted to non-commercial use.Source: §5 · p. 4
• Among freely usable models, omniASR LLM 1B v2 and Whisper-large-v3 lead on accuracy, with omniASR LLM 1B v2 most robust across low-resource and typologically diverse languages.Source: §5 · p. 4
• RTFx 103 gives omniASR CTC 1B v2 the best accuracy–efficiency trade-off, while parakeet-tdt-0.6b-v3 reaches RTFx 152 at substantially degraded WER.Source: §5 · p. 4
• High-resource German, Spanish, and Portuguese show consistent performance, whereas lower-resource languages—especially Basque—expose sharp divergences and severe degradation for models without coverage.Source: §5 · p. 4
• Speaker-sex disaggregation shows a consistent male advantage, with only Scribe v2 and seamlessm4t-v2-large remaining equitable.Source: §5 · p. 4