Source-linked AI summary
SEA-HELM Audio: Evaluating Multilingual ASR-LLM Pipeline Performance Across SEA-HELM Subsets
Aulia Adila, Yixian Tan, Jonathan Heng, Kunat Pipatanakul, Donghang Wu, Anocha Sutaveephamochanon
TL;DR
Evidence on voice-agent performance across Southeast Asian languages is limited, despite text benchmarks’ strong scores and the potential for speech input to alter task performance. SEA-HELM Audio compares direct text input with synthesized-speech pipelines across five languages. The modality gap varies by language and task, with Chinese multiple-choice reasoning showing under 11 points of degradation and Tamil instruction following exceeding 40 points.
Problem
Existing spoken-language benchmarks are predominantly English-language, leaving performance across Southeast Asian languages and the downstream effects of transcription errors insufficiently understood.
Source: §1 · p. 1Method
SEA-HELM Audio adapts regional text prompts into synthesized-speech pipelines and compares speech and text accuracy using the same LLM and verifier.
Source: §1 · p. 1, §3.1 · p. 2Results
Chinese multiple-choice reasoning exhibited a modality gap below 11 points, while Tamil instruction following had a gap above 40 points.
Source: §6 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
We introduce SEA-HELM Audio, a Southeast Asian (SEA) multilingual evaluation framework that extends the text-based SEA-HELM benchmark to assess performance degradation in cascaded automatic speech recognition and large language model (ASR-LLM) pipelines. Using synthesized speech, we compare speech-input and text-input performance across five languages (Chinese, English, Malay, Indonesian, and Tamil) on multiple-choice reasoning and instruction-following tasks. We evaluate open-weight and commercial ASR-LLM systems using identical verifiers and quantify the speech gap as the difference between text and speech accuracy. Results show substantial degradation, with accuracy drops of 9.00 to 18.50 percentage points for multiple-choice reasoning and 20.00 to 42.86 points for instruction following. Tamil exhibits the largest degradation, exceeding 40 percentage points. Our findings highlight the limitations of text-only benchmarks in predicting voice-agent reliability and demonstrate the value of SEA-HELM Audio for evaluating multilingual voice-agent systems.
1. Introduction
SEA-HELM Audio addresses the limited evidence on how speech pipelines perform across Southeast Asian languages and tasks. It adapts SEA-HELM’s text prompts into synthesized speech evaluations to measure the accuracy difference between speaking and typing.
1. Introduction
- As AI moves toward voice-enabled agents, speech pipelines introduce acoustic transcription errors, phonetic ambiguities, and domain-specific degradation that can affect end-to-end accuracy.The paper motivates measuring the accuracy penalty users incur by speaking rather than typing.Source: §1 · p. 1
- Existing spoken-language benchmarks such as VoiceBench [1] are predominantly English-language, leaving Southeast Asian voice-agent performance less understood.The extent to which transcription errors alter task-critical information and downstream performance is also unclear.Source: §1 · p. 1
- SEA-HELM provides regional prompts and tailored programmatic verifiers, but was built as a text-only benchmark without paired audio.Source: §1 · p. 1
- The speech gap measures the difference between text and speech accuracy in a speech-based adaptation of SEA-HELM [2].The framework compares synthesized-audio pipelines with the benchmark’s original text prompts.Source: §1 · p. 1
2. Methodology
The methodology adapts SEA-HELM tasks for controlled, deterministic comparison of text and synthesized-speech inputs across five languages. It selects speakable prompts with rule-based verifiers and defines the speech penalty as the difference between text and speech accuracy.
2.1. Experimental Design
- The experimental design evaluates each regional prompt both through synthesized speech and directly as text to isolate the speech-to-text layer’s performance penalty.The prompts preserve regional nuances, code-switching patterns, and local cultural contexts across the five languages.Source: §2.1 · p. 1
2.2.1. Benchmark Alignment and Active Subset Scope
- The active evaluation focuses on SEA-HELM IFEval and Global-MMLU-Lite across Chinese, Malay, Tamil, Indonesian, and English.The broader evaluation suite draws from four primary SEA-HELM task subsets.Source: §2.2.1 · p. 2, §2.2 · p. 2
2.2. Datasets and Task Selection
- Tasks were selected for speakable prompts and programmatic rule-based verifiers, ensuring identical scoring on text and speech paths.The selected task formats are multiple-choice and instruction following, instantiated with SEA-HELM prompts.Source: §2.1 · p. 1, §2.2.1 · p. 2
2.3. Evaluation Metrics
- The study measures the modality penalty as text accuracy minus end-to-end speech accuracy across each benchmark run.The text score is the target LLM’s accuracy on the original prompt; the speech score follows TTS synthesis and ASR transcription.Source: §2.3 · p. 2
3. SEA-HELM Audio
SEA-HELM Audio turns written benchmark prompts into a speech path and compares its outcomes with direct text input using the same LLM and verifier. The pipeline synthesizes only user-spoken content, transcribes it with ASR, and scores outputs with deterministic settings and rules.
3. SEA-HELM Audio
- SEA-HELM Audio supplies spoken inputs for a text-only benchmark by synthesizing each prompt, transcribing it, and sending the transcript to an LLM.The LLM’s response to the original text prompt provides the comparison path.Source: §3 · p. 2
3.1. Evaluation Flow
- The speech path uses TTS, ASR, and an LLM, while the text path sends the original prompt directly to that same LLM; a shared verifier scores both.Their accuracy difference is the accuracy lost by speaking instead of typing.Source: §3.1 · p. 2
3.2. Spoken Prompt Construction
- Only content a user would say is synthesized; answer keys and instruction-following constraints remain unspoken and are used at scoring time.For Chinese multiple-choice prompts, option labels and the answer marker are rendered as Chinese words.Source: §3.2 · p. 2
3.3. Speech Recognition
- Three ASR systems cover target languages, and transcription changes are measured separately with Word Error Rate and Character Error Rate.The systems are Qwen3-ASR-1.7B [7], MERaLiON3-ASR-3B [8], and ElevenLabs Scribe Realtime; open-weight models use greedy decoding.Source: §3.3 · p. 2
3.4. Response Generation
- SEA-LION-4.5-27B [9] and Gemini 3.5 Flash answer both the ASR transcript and original prompt, using temperature 0 for deterministic decoding.This holds the LLM and decoding behavior constant across the input paths.Source: §3.4 · p. 2
3.5. Scoring
- Rule-based SEA-HELM verifiers score both paths identically, checking answer options for multiple-choice tasks and prompt constraints plus expected language for instruction following.Source: §3.5 · p. 2
4. Results
Results compare text and speech accuracy across multiple-choice reasoning and instruction following, with larger modality gaps in instruction following and especially steep degradation for Tamil. On Chinese and English multiple-choice reasoning, the commercial stack outperformed the regional open-weight baseline in both modalities.
4.1. Multiple Choice Reasoning (Global-MMLU-Lite)
- Across Chinese and English Global-MMLU-Lite evaluations, the commercial stack outperformed the regional open-weight baseline in both text and speech accuracy.Each run used 400 samples.Source: §4.1 · p. 2, Table 3 · p. 3
- Chinese multiple-choice gaps were 9.00–10.75 points, while English gaps were 15.50–18.50 points.Chinese showed greater resilience to the shift from text to speech than English.Source: §4.1 · pp. 2–3, Table 3 · p. 3
4.2. Instruction Following (SEA-HELM IFEval)
- IFEval tested deterministic instruction adherence in English, Malay, Indonesian, and Tamil; text-only accuracy peaked at 97.14% for English, but speech caused severe degradation in every tested language.Each run used 105 samples.Source: §4.2 · p. 3, Table 4 · p. 3
- Instruction-following modality gaps expanded significantly versus multiple-choice evaluations, with Tamil drops exceeding 40 points for both evaluated pipelines.The reported Tamil result indicates vulnerabilities in ASR transcriptions.Source: §4.2 · p. 3, Table 4 · p. 3
- SEA-LION-4.5-27B surpassed Gemini 3.5 Flash in Indonesian IFEval text accuracy, 93.33% vs 89.52%, yet its open-weight pipeline had gaps over 25 points in Malay and Indonesian.The commercial stack generally outperformed the regional open-weight baseline.Source: §4.2 · p. 3
5. Discussion & Analysis
Speech accuracy varies substantially with language, task structure, and pipeline composition. Chinese is relatively resilient, while Tamil instruction following and regional open-weight ASR pipelines show pronounced weaknesses.
5. Discussion & Analysis
- The modality gap varies with language, task structure, and pipeline composition, so strong text-based LLM performance does not guarantee equivalent voice capability.The discussion frames these disparities as relevant to voice-agent deployment.Source: §5 · p. 3
5.1. Linguistic Resilience vs. Vulnerability
- Chinese lost approximately 9.00 to 10.75 points, while Tamil lost over 40 despite SEA-LION-4.5-27B's 85.71% text baseline and 42.86% speech accuracy.The discussion attributes the Tamil bottleneck to acoustic processing and identifies data scarcity in Tamil ASR training corpora as a concern.Source: §5.1 · p. 4
5.2. Task Format Sensitivity: MCQ vs. Instruction Following
- IFEval gaps were 20 to 43 points, compared with greater recovery on multiple-choice tasks where contextual clues can compensate for a misheard noun.IFEval's lexical, length, and format constraints can be lost when ASR drops or substitutes a prompt word, causing the verifier to mark an otherwise correct response as a failure.Source: §5.2 · p. 4
5.3. Commercial vs. Open-Weight
- The commercial stack had higher speech accuracy, but SEA-LION-4.5-27B beat Gemini on Indonesian IFEval text accuracy, 94.29% vs. 89.52%, before a 33.33-point voice penalty.The discussion identifies fragility in the open-weight ASR models handling Southeast Asian accents and code-switching, rather than regional LLM reasoning, as the limiting factor.Source: §5.3 · p. 4
6. Conclusion and Future Work
The study adapts a text-native benchmark to evaluate speech pipelines on deterministic tasks across five languages. Its results show that voice reliability depends on language and task rigidity, with open-weight deployment bottlenecks attributed to acoustic transcription.
6. Conclusion and Future Work
- SEA-HELM, a text-native benchmark, can be adapted to assess end-to-end voice pipelines when evaluation tasks are highly deterministic.The study measures the gap between pure-text and speech accuracy across five languages.Source: §6 · p. 4
- Voice reliability varies with language and task rigidity: Chinese multiple-choice reasoning had ∆< 11 points, while Tamil instruction following had ∆> 40 points.The commercial stack provided the most stable end-to-end performance.Source: §6 · p. 4
- SEA-LION-4.5-27B's strong pure-text baseline points to acoustic transcription, rather than language-model reasoning, as the primary bottleneck for regional open-weight deployments.The conclusion contrasts pure-text capability with end-to-end voice performance.Source: §6 · p. 4