Source-linked AI summary
SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia
Jingyi Liao, Wenyu Zhang, Zhuohan Liu, Yingxu He, Geyu Lin, Xunlong Zou, Shuo Sun, Syed Ali Redha Alsagoff, Ai Ti Aw
TL;DR
Speech-understanding evaluation has largely overlooked Southeast Asian languages, despite their linguistic diversity and large speaker population. SEA-SpeechBench addresses this gap with a multilingual, multitask benchmark spanning curated datasets, temporal reasoning, and model evaluations. Results reveal systematic weaknesses in temporal reasoning, emotion recognition, and speech translation, alongside English-prompt advantages and low-resource-language disparities.
Problem
Existing speech-understanding benchmarks overlook Southeast Asian languages, making standardized evaluation across the region difficult.
Method
SEA-SpeechBench evaluates 11 SEA languages across 9 tasks using curated data, temporal reasoning tasks, multilingual prompts, and standardized comparisons.
Results
Evaluation exposes systematic weaknesses in temporal reasoning, emotion recognition, and speech translation, with English prompts consistently outperforming native languages.
Takeaways & Limitations
The benchmark establishes a baseline for developing temporally robust, linguistically inclusive, and practically deployable speech technologies for Southeast Asia.
Takeaways & Limitations
Coverage of some dialects and task categories remains limited because high-quality annotated resources are scarce.
Abstract
from arXiv · showhide
The rapid advancement of audio and multimodal large language models has unlocked transformative speech understanding capabilities, yet evaluation frameworks remain predominantly English-centric, leaving Southeast Asian (SEA) languages critically underrepresented. We introduce SEA-SpeechBench, to the best of our knowledge, the first large-scale multitask benchmark that evaluates speech understanding in 11 SEA languages through 97,194 samples across 99 evaluation sets and 597 hours of curated audio data. Our benchmark comprises 9 diverse tasks across 3 categories: speech processing (automatic speech recognition, speech translation, spoken question answering), paralinguistic analysis (emotion, gender, age, speaker recognition), and temporal understanding, a novel dimension featuring timestamped content queries and temporal localization within extended audio sequences up to 3 minutes. We implement multilingual prompting in both native SEA languages and English to reflect user interactions with audio-language models. Evaluation of leading open-source and proprietary systems reveals marked performance gaps. Across all models, performance remains underwhelming on temporal understanding, emotion recognition, and speech translation. Prompting in low-resource languages such as Burmese and Tamil lags behind English by up to 41 percentage points. Our findings expose critical model limitations and underscore the need for inclusive model development. The SEA-SpeechBench benchmark is available at https://zwenyu.github.io/SEA-SpeechBench/.
1 Introduction
SEA-SpeechBench addresses the underrepresentation of Southeast Asian languages in speech-understanding evaluation with a unified, multilingual benchmark. It combines broad regional coverage with temporal reasoning tasks and evaluations of open-source and proprietary models.
- Existing speech-understanding benchmarks largely overlook Southeast Asian languages despite the region representing over 650 million speakers.
- Unified task definitions, prompting, and normalization are needed because heterogeneous SEA datasets make standardized comparison difficult.
- SEA-SpeechBench evaluates 11 SEA languages across 9 tasks in speech processing, paralinguistics, and temporal reasoning.The benchmark contains more than 97,000 audio samples and over 597 hours of audio data.
- Temporal reasoning tasks assess content retrieval and localization within extended audio streams, addressing an unexplored dimension of audio-LLM evaluation.
- Each task is evaluated with both English and native-language text prompts to reflect authentic usage scenarios.
2 SEA-SPEECHBENCH Evaluation Suite
The evaluation suite organizes nine audio-understanding tasks across speech processing, paralinguistic analysis, and temporal reasoning, using curated multilingual data and standardized task-specific evaluation. Its design combines short clips with recordings up to three minutes and uses deterministic metrics or validated LLM judges where needed.
- Task suite: SEA-SpeechBench requires textual responses to audio inputs and text queries across nine tasks in three categories.
- Speech processing: Speech processing covers ASR, speech translation from SEA languages to English, and spoken question answering.
- Paralinguistic analysis: Paralinguistic analysis includes emotion, gender, age, and speaker recognition based on vocal cues or speaker metadata.
- Temporal reasoning: Temporal reasoning maps timestamp intervals to speech content and content queries to temporal locations in recordings up to 3 minutes long.
- Data curation: Source selection is task-specific: reliable transcripts support ASR and translation, while metadata and timestamp alignments support paralinguistic and temporal tasks.
- Data curation: The benchmark combines short utterances with stratified long-audio bins of 30–60s, 60–120s, and 120–180s, totaling over 10,000 long-audio samples.
- Metrics: Metrics include WER/CER, BLEU, chrF, span-overlap F1, macro-F1, and LLM-based judgments for free-form SQA and emotion outputs.
- Metrics: GPT-4.1 judges SQA and Gemma-3-27B-Instruct judges emotion recognition after comparison with human annotations.
3 Evaluations
The evaluation compares open-source and commercial audio/multimodal models across speech processing, paralinguistics, temporal reasoning, and prompt languages. Results reveal weak translation, temporal reasoning, and native-language prompting, alongside refusal-driven speaker-recognition failures.
- Speech Processing: Gemini 2.5 Flash achieves the best overall ASR performance, while MERaLiON-2 leads among open-source models.Qwen3-Omni is competitive on Vietnamese, Indonesian, and Thai, but Tamil and Burmese remain particularly difficult.
- Speech Processing: 21 BLEU and 45–47 chrF are the open-source ceilings for speech translation, below commercial systems reaching 48–54 chrF.Gemini 2.5 Flash reaches 53.94 chrF under SEA prompts but only 18.89 BLEU, showing metric divergence.
- Temporal Reasoning: Temporal reasoning degrades as audio duration increases, with errors accumulating through boundary drift and truncation.Coverage exceeds purity across durations, indicating over-coverage and weak temporal boundary localization.
- Prompt Language: English prompts outperform native SEA prompts most strongly for Myanmar, Tamil, and Khmer, with English advantages ranging from −19.5 to −41.2 PAS.Prompt-language choice is near-neutral for Chinese, English, and Indonesian, while Filipino, Vietnamese, Malay, Lao, and Thai show moderate English preference.
- Paralinguistic Analysis: GPT-4o records 14.49% speaker-recognition accuracy because it refuses 89.5% of queries, while answered queries are 100% correct.This pattern is consistent with deliberate abstention rather than task incomprehension; Qwen2.5-Omni-7B also refuses frequently but makes mistakes when answering.
4 Related Works
Recent audio-language models connect speech encoders and language models through adaptors, shared token spaces, and stronger multimodal or temporal modeling. Benchmarking has expanded beyond recognition toward instruction following and reasoning.
- Audio/Multimodal LLMs and Benchmarks: Audio-language models use lightweight adaptors, shared speech-text token spaces, or stronger multimodal and temporal modeling.The related systems include Phi-4, Qwen2.5-Omni, Voxtral, GPT-4o, and Gemini.
- Audio/Multimodal LLMs and Benchmarks: Evaluation has broadened from speech recognition to instruction following and reasoning through newer audio-language benchmarks.
5 Conclusion
SEA-SPEECHBENCH provides a standardized, bilingual benchmark for speech understanding across 11 Southeast Asian languages and nine tasks. Its evaluation exposes temporal brittleness, linguistic inequity, and persistent weaknesses in temporal reasoning, emotion recognition, and speech translation.
- Benchmark Contribution: SEA-SPEECHBENCH covers 11 Southeast Asian languages and 97,000+ samples across nine tasks spanning speech processing, paralinguistics, and temporal reasoning.
- Benchmark Contribution: Unified normalization and bilingual prompting enable reproducible cross-linguistic comparisons.
- Main Findings: Evaluation exposes long-audio performance collapse, English advantages over native prompts, and low performance in temporal reasoning, emotion recognition, and speech translation.The authors characterize these findings as systematic weaknesses and persistent scalability and generalization gaps.
6 Limitations and Risks
The benchmark is constrained by limited data availability and potential test-set contamination. These risks restrict dialectal and task-level coverage and complicate interpretation when model training data are undisclosed.
- Data Availability: Limited high-quality annotated resources constrain coverage of certain dialects and task categories.The resulting limitation restricts the breadth of dialectal and task-level evaluation that can be reliably supported.
- Test-Set Contamination: Undisclosed model training data and curation pipelines make test-set contamination difficult to control.
A Dataset Catalog
SEA-SpeechBench standardizes diverse speech datasets into a reproducible benchmark while preserving task-specific metadata, label, and class-distribution decisions. Its catalog spans source coverage, licensing, multilingual prompts, sampling, and evaluation safeguards.
- Metadata is used selectively for labels, references, filtering, pairing, timestamp construction, and split enforcement when reliable for the target evaluation.
- YODAS2 transcripts support filtered SQA, TCQ, and TLoc construction but are excluded as gold ASR references because utterance-level reliability is inconsistent.
- SEA-SpeechBench catalogs source datasets by task, language coverage, audio duration, and licensing information.
- The benchmark standardizes audio, prompts, sampling, identifiers, and filtering to create a unified, reproducible evaluation artifact.
- Emotion labels are normalized into a closed nine-class taxonomy through source mapping and canonical lexical normalization.
- AgeR has normalized entropy ˜H = 0.745, while GR, SpkR, and ER are near-uniform; macro-F1 is therefore used for GR, SpkR, and AgeR.
B.3 New Evaluation Data Construction
The benchmark constructs new speaker, question-answering, and temporal datasets through controlled sampling, transcript filtering, alignment checks, and standardized prompts. It also defines Prompt Advantage Score to quantify both the direction and magnitude of language-prompt effects.
- SpkR concatenates same-speaker or different-speaker audio pairs separated by a beep to form controlled speaker-verification instances.
- SQA uses filtered YODAS2 transcripts to generate question-answer pairs, followed by native-speaker audits for grounding, correctness, and question diversity.
- SQA model rankings show high judge agreement, with Spearman ρ = 0.967 for English prompts and ρ = 0.885 for SEA prompts.
- TCQ asks models to transcribe speech between timestamps, whereas TLoc asks them to predict the start and end timestamps of specified utterances.
- Prompt Advantage Score combines prompt-language direction and effect size, with positive values favoring local prompts and negative values favoring English.
- For lower-is-better ASR and TCQ metrics, scores are transformed before aggregation so positive PAS consistently denotes local-prompt advantage.
D Text-based Language Understanding
The prompt-only sanity check examines whether multilingual performance gaps reflect text-based language understanding alone. Its findings instead point to a broader cross-modal instruction-following or alignment gap under non-English prompts.
- The study tests AgeR prompts in Tamil, Thai, Vietnamese, and Chinese by asking models to translate each prompt and describe the required output.
- The observed prompt asymmetry cannot be attributed solely to language understanding and instead points to a broader cross-modal instruction-following or alignment gap.
- Table 15 defines positive ∆ as English prompts outperforming native-language prompts and reports prompt-check failure rates.
E Postprocessing of Evaluation
Postprocessing standardizes textual evaluation inputs through Unicode, case, punctuation, and prompt normalization. The benchmark evaluates with parallel English and native-language prompts under a consistent judging protocol.
- The text normalization pipeline applies NFC Unicode normalization, lowercasing, and configurable punctuation filtering to reduce encoding and case inconsistencies.
- Normalization heuristics are validated and refined across Southeast Asian languages.
- Evaluation uses parallel prompts in native SEA languages and English, with examples covering the benchmark’s task families.
- Model-as-judge prompts are adopted from prior work, with human judges following the same evaluation instruction for protocol consistency.
H Detailed Evaluation Results
This section reports detailed comparisons between English and SEA prompts across speech, paralinguistic, translation, and temporal-understanding evaluations. Metrics use task-specific scoring conventions, with lower values better for ASR and TCQ and higher values better for the remaining reported metrics.
- ASR: ASR results compare English and SEA prompts using raw WER and CER, where lower scores indicate better performance.The reported values are not normalized.
- Paralinguistic analysis: Age, emotion, gender, and speaker recognition results compare English and SEA prompts using macro-F1 or Model-as-Judge scores, with higher values indicating better performance.Age, gender, and speaker recognition use macro-F1, while emotion recognition uses Model-as-Judge.
- Speech translation: Speech translation results compare English and SEA prompts using BLEU and chrF, with higher scores indicating better performance.The tables report separate BLEU- and chrF-based evaluations.
- Temporal understanding: TCQ results use raw WER and CER, whereas TLoc results use F1, with lower scores preferred for TCQ and higher scores preferred for TLoc.Both temporal-understanding tasks compare English and SEA prompts across datasets.