Source-linked AI summary
Nuha-Speech: Building General-Purpose Arabic Speech-LLMs
Yingzhi Wang, Reem Alhazzani, Muhammad Alqurishi
TL;DR
Arabic speech-LLMs lack public training resources and a broadly adopted evaluation benchmark, while existing support and evidence remain limited. Nuha-Speech addresses this gap with a large, reproducibly curated instruction corpus, Qwen-Omni fine-tuning, and a multi-task Arabic benchmark. Fine-tuning improves performance across the listed tasks and establishes infrastructure for future Arabic speech-LLM research.
Problem
Arabic speech-LLM development is constrained by scarce public non-ASR corpora, extremely limited genuine Arabic speech-instruction data, and the absence of a widely adopted Arabic evaluation benchmark.
Method
Nuha-Speech combines mostly public and curated datasets, instruction-tunes three Qwen-Omni variants, and evaluates Arabic speech understanding and paralinguistic tasks with task-specific metrics.
Results
Fine-tuned models achieved clear improvements across all listed tasks, with larger gains for the 3B and 7B models and for paralinguistic tasks.
Takeaways & Limitations
Nuha-Speech provides unified Arabic support across broad speech tasks and establishes a common benchmark for future Arabic speech-LLM research.
Takeaways & Limitations
Reliance on predominantly private datasets can restrict reproducibility and broader community adoption.
Abstract
from arXiv · showhide
As Speech Large Language Models (speech-LLMs) become increasingly multilingual, Arabic remains significantly underrepresented, highlighting the need for dedicated infrastructure to train and evaluate Arabic speech-LLMs. To address this gap, we introduce Nuha-Speech, a comprehensive initiative to develop general-purpose Arabic speech-LLMs spanning dataset construction, model training, and systematic evaluation. Specifically, we constructed a large-scale Arabic Speech Question-Answering (SQA) corpus comprising over 1.5 million training samples to allow instruction tuning over a broad range of core speech tasks. Then, the corpus was used for supervised fine-tuning based on Qwen-Omni model variants at different scales. Finally, we designed an evaluation framework featuring diverse tasks and tailored metrics. Through this work, we aim to establish foundational infrastructures for Arabic Speech-LLMs under constraints imposed by limited Arabic speech resources.
1 Introduction
Arabic remains underrepresented in speech-LLMs, with scarce public speech-instruction data and no widely adopted Arabic evaluation benchmark. Nuha-Speech addresses these gaps through a large training corpus, Qwen-Omni fine-tuning, and multi-task evaluation.
- Research Gap: Arabic speech-LLMs lack sufficient public corpora for non-ASR tasks and genuine Arabic speech-instruction pairs remain almost non-existent.The scarcity is especially acute for Speech Emotion Recognition and Speech Question Answering.
- Research Gap: Arabic speech-LLM evaluation lacks a widely adopted benchmark because existing benchmarks and tasks are heavily centered on English.This makes assessment of Arabic understanding difficult.
- Nuha-Speech: Nuha-Speech combines dataset construction, model training, and benchmark development to build general-purpose Arabic speech-LLMs.The initiative uses public and curated data, fine-tunes three Qwen-Omni variants, and evaluates models before and after fine-tuning.
- Nuha-Speech: The training corpus contains 1.5 million Arabic speech instruction-following samples spanning speech understanding and paralinguistic tasks.The authors emphasize mostly public datasets and transparent curation to facilitate replication.
- Results: Fine-tuned models achieved clear improvements across all listed tasks compared with their pre-finetuning settings.This result is reported across the multi-task evaluation framework.
2 Nuha-Speech Dataset
Nuha-Speech unifies diverse Arabic speech tasks in a shared text-output training format. Its corpus combines large public ASR resources with task-specific datasets and broad instruction sets designed to support generalization.
- Corpus Design: Each training example uses a shared {speech, instruction, output} format with text-only outputs across tasks.The corpus also uses broad GPT-5-generated instruction sets to support zero-shot generalization and reduce overfitting to narrow prompts.
- Corpus Design: The corpus overview organizes collected datasets by task, source, and size.Table 1 provides the corresponding training-corpus overview.
- Automatic Speech Recognition (ASR): ASR is treated as the primary objective and draws on four large-scale, multi-dialect public datasets.The selected resources are SADA, Common Voice, MASC, and MGB-2.
- Automatic Speech Recognition (ASR): 790K ASR training samples remain after filtering segments to the 1–10s duration range.The filtering is described as an efficiency-oriented preprocessing step.
2.2 Automatic Speech Translation (AST)
Nuha-Speech trains Arabic-to-English speech translation using an existing CoVoST-v2 split and additional translated ASR data. The augmentation addresses the limited size of available AST samples.
- Automatic Speech Translation (AST): Arabic-to-English AST maps spoken Arabic input directly to English textual translation in an end-to-end setup.The paper frames this direction as valuable for Arabic workflows.
- Automatic Speech Translation (AST): The AST corpus uses the Arabic-to-English split of CoVoST-v2, supplemented because its sample count is very limited.The passage identifies CoVoST-v2 as the widely used source dataset.
- Automatic Speech Translation (AST): 150K additional AST samples were created by translating randomly selected ASR transcriptions into English with Qwen3-32B.The augmentation draws from the previously collected ASR dataset.
2.3 Speech Question Answering (SQA)
Nuha-Speech constructs Arabic SQA data from MGB-2 transcriptions because no suitable dataset combines spoken context with textual question-answer pairs. It generates both closed- and open-ended examples, while also incorporating dialect identification as a core Arabic speech task.
- Speech Question Answering (SQA): Arabic SQA lacks a suitable dataset containing both spoken context and textual question-answer pairs.The authors therefore use MGB-2 transcriptions as the source for synthetic QA construction.
- Speech Question Answering (SQA): 300K SQA pairs are generated from MGB-2 transcriptions, split evenly between closed-ended and open-ended questions.The closed-ended set targets structured queries and exact retrieval, whereas the open-ended set targets cross-sentence reasoning and expressive generation.
- Dialect Identification (DI): Dialect identification is included because Arabic speech technologies must handle substantial dialectal diversity.The ADI-17 dataset supplies samples for classifying 17 Arabic dialects.
2.5 Speech Emotion Recognition (SER)
Nuha-Speech expands Arabic speech emotion recognition data through balanced synthetic samples spanning four emotion categories and multiple speakers. It also addresses age-coverage gaps by combining underrepresented age groups with balanced adult samples.
- 20K emotional samples were synthesized across balanced angry, happy, sad, and neutral categories using eight speakers per category.
- The emotion data used multiple speakers per category to reduce the risk of speaker-specific bias.
- 10K age-classification samples were built for young, adult, and elder groups by combining SADA and Common Voice samples with balanced adult additions.
- An additional 10K samples were synthesized with 10 distinct speakers per age group using the same ElevenLabs TTS model.
2.7 Gender Recognition (GR)
Nuha-Speech includes gender recognition training data to address cross-lingual variability in Arabic gender-recognition performance.
- 51K gender-recognition samples were extracted from SADA with equal representation of male and female speakers.
- Gender recognition was included because prior work observed cross-lingual variability in recognition performance.
2.8 Multi-SQA
Nuha-Speech adds multi-turn SQA tasks that combine speech attributes, transcription, and cross-modal emotion reasoning. These tasks are designed to strengthen robustness and adaptability in complex spoken-language scenarios.
- A 5K multi-turn task merges gender, age, and ASR annotations from SADA, with balanced gender and age-group distributions.
- The multi-turn SQA additions target model robustness and adaptability in complex spoken-language scenarios.
- A 10K-sample multi-turn task asks models to recognize emotion, transcribe speech, and infer emotion correlation across the two modalities.
3 Instruction Tuning
Nuha-Speech instruction tuning uses Qwen-Omni baselines at multiple scales and a two-stage curriculum that first establishes ASR before training across tasks. The resulting LoRA-finetuned models are named Nuha-Speech-3B, 7B, and 30B.
- 3.1 Baseline Models: Nuha-Speech fine-tunes Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Qwen3-Omni-30B as baseline models.
- 3.1 Baseline Models: Qwen2.5-Omni uses a Whisper-large-v3 encoder, whereas Qwen3 uses an AuT audio encoder and a Mixture-of-Experts architecture.
- 3.2 Experimental Setups: Table 2 presents the Nuha-Speech training curriculum used for instruction tuning.
- 3.2 Experimental Setups: The two-stage curriculum first trains ASR for audio-text mapping, then trains on all tasks while reducing ASR samples from 790K to a random 200K subset.
- 3.2 Experimental Setups: Qwen2.5-Omni models use global batch size 16 and learning rate 1e-4, while Qwen3-Omni-30B uses batch size 8 at the same learning rate.
- 3.2 Experimental Setups: LoRA uses rank 8 and alpha 32 on all linear layers, and the Qwen3 MoE router uses a load-balancing loss coefficient of 1e-3.
- 3.2 Experimental Setups: The fine-tuned models are named Nuha-Speech-3B, Nuha-Speech-7B, and Nuha-Speech-30B according to their baseline parameter sizes.
4 Nuha-Speech Benchmark
Nuha-Speech evaluates Arabic speech-LLMs across seven speech understanding and paralinguistic tasks using curated test sets, task-specific metrics, and baseline comparisons. Fine-tuning improves all evaluated models, with especially large gains on paralinguistic tasks and smaller models.
- 4.1 Test Sets and Metrics: The evaluation uses seven training-aligned tasks spanning speech understanding and paralinguistics, with datasets, metrics, and sample counts summarized in Table 3.The study reports average WER for ASR and accuracy for four paralinguistic tasks, with additional metrics for AST and SQA.
- 4.2 Results: Fine-tuning improved every evaluated model across all seven tasks, with larger gains for 3B and 7B models and smaller gains for the already-strong 30B model.The benchmark compares original baselines with Nuha-Speech fine-tuned versions.
- 4.2 Results: Paralinguistic tasks gained more substantially than semantic tasks after fine-tuning, suggesting Arabic speech paralinguistics were under-represented during pretraining.
- 4.2 Results: Nuha-Speech-30B consistently achieved the best AST performance, while Nuha-Speech-7B marginally outperformed the other fine-tuned models on SQA.The ASR capability of Nuha-Speech-30B remained strong, whereas the 3B and 7B models improved substantially over their baselines.
- 4.2 Results: Dialect identification and age recognition rose multifold after fine-tuning, with Nuha-Speech-30B remaining the top-performing model.Their baselines initially performed close to random guessing.
- 4.2 Results: Qwen2-Audio-7B-Instruct underperformed the same-sized Qwen2.5-Omni-7B after fine-tuning and was excluded from the final benchmark.The authors retained a more competitive baseline set for the final comparison.
5 Conclusions
Nuha-Speech provides general-purpose Arabic speech-LLMs with unified instruction-following across broad speech tasks. It combines largely open-access data, documented fine-tuning protocols, and a common benchmark to support reproducible Arabic speech-LLM research.
- 5 Conclusions: Nuha-Speech provides unified Arabic instruction-following across broad speech understanding and paralinguistic tasks.
- 5 Conclusions: The work promotes reproducibility through largely open-access datasets, transparent curation strategies, and fully documented fine-tuning protocols.
- 5 Conclusions: Its common benchmark jointly evaluates Arabic speech understanding and speech paralinguistics capabilities, laying infrastructure for future general-purpose Arabic speech-LLM development.