PaperlayerView exact revision on arXiv

Source-linked AI summary

Developing a Speech Language Model for Basque

Andoni Sudupe, Inma Hernaez, Ander Arriandiaga, Christoforos Souganidis, Ibon Saratxaga, Eva Navas

arXiv:2610.36911v1cs.CLcs.AIeess.AS

TL;DR

Open spoken-language systems and spoken-instruction data are scarce for minority languages such as Basque. This work builds a Basque SLM using modular speech components and translated, synthesized instruction data, reporting content-quality and intelligibility results.

  • Problem

    Open SLMs remain focused almost exclusively on English and Chinese, and Basque lacks public spoken-instruction triplets for SLM training.

    Source: §1 · p. 1, §3.3 · p. 3

  • Method

    The authors adapt a Latxa-based SLM with speech modules and create Basque training data by translating English instructions and synthesizing their audio.

    Source: §1 · p. 1

  • Results

    The model scored 3.86 out of 5 in content quality and achieved 9.94% WER for Basque words, with 370 ms streaming latency.

    Source: §6 · p. 4

  • Takeaways & Limitations

    The work establishes a reproducible open-source Basque SLM framework using modular components and synthetic data without massive foundational pre-training.

    Source: §6 · p. 4

    Further work is planned on expressive TTS, a second judge and small-scale human evaluation, and testing with real, nonsynthetic Basque speech.

    Source: §6 · p. 4

Abstract

from arXiv · show

Although the recent development of LLMs has paved the way for multi-modal interaction, minority languages face difficulties keeping pace. This work presents the first open Speech Language Model for Basque, designed for low-latency, end-to-end spoken interaction. Based on the LLaMA-Omni architecture, the system integrates a Latxa model as its backbone, transferring its Basque capabilities to the spoken modality by fine-tuning several modules. To address data scarcity, the English VoiceAssistant-400K corpus was translated into Basque, and the audio was synthesized using a Basque TTS. Evaluation results confirm the system's viability, achieving 3.86/5 in content quality, a 9.94% WER for spoken intelligibility, and a 3.07 UTMOS for audio quality. This work demonstrates that advanced spoken interaction systems can be developed using modular architectures and synthetic data even in low-resource languages.

1. Introduction

The paper addresses the lack of open spoken-language systems and suitable speech-instruction data for Basque by adapting a language-specific model with synthetic data. It reports content-quality and intelligibility results for the resulting system.

1. Introduction

  • Minority-language SLMs face a gap because open-weight systems remain focused almost exclusively on English and Chinese, while speech data must belong to the target language.The paper identifies LLaMA-Omni, LLaMA-Omni2 [2], and Spirit LM [3] as examples.Source: §1 · p. 1
  • The system follows LLaMA-Omni, using a Whisper encoder and a Unit HiFi-GAN vocoder trained from HuBERT tokens to generate speech.The study investigates development, training recipes, and evaluation methods for an open Basque SLM.Source: §1 · p. 1
  • The authors translated English VoiceAssistant-400K [7] into Basque with Latxa-3.1-70B-Instruct [8] and synthesized the corresponding audio with Basque TTS.They describe this as the first multimodal spoken-instruction dataset for Basque.Source: §1 · p. 1
  • 3.86 out of 5 in content quality and 9.94% normalized WER for Basque words support the reported viability of synthetic-data training and modular transfer.The content-quality score is based on LLM-as-a-Judge evaluation; the WER is reported as comparable to the original LLaMA-Omni model's English intelligibility.Source: §1 · p. 1

2. System Architecture

The system attaches speech-processing modules to a Latxa language backbone. It integrates speech input into the LLM and generates text and speech in parallel for low-latency response.

2. System Architecture

  • A speech adaptor maps encoded speech into the LLM's embedding space, downsampling and projecting features through a two-layer FFN with ReLU.The architecture is based on LLaMA-Omni.Source: §2 · p. 2, Figure 1 · p. 2
  • The LLM's hidden states feed parallel text and speech decoding, avoiding intermediate-text dependence; a non-autoregressive Transformer predicts discrete speech tokens for the vocoder.During streaming inference, generated tokens are sent to the vocoder in chunks for immediate playback.Source: §2 · p. 2
  • Latxa-3.1-8B-Instruct supplies the Basque backbone, Whisper-large-v3-eu encodes input speech, and mHuBERT-147-derived discrete tokens feed a Unit HiFi-GAN vocoder.The discrete-token encoder is fine-tuned for ASR and clustered into 1,000 units; the vocoder generates waveforms from those units.Source: §2 · p. 2

3. Data

The data pipeline supplies distinct training sets for the speech encoder, vocoder, and SLM. Because Basque spoken-instruction triplets were unavailable, the authors generated a large synthetic instruction corpus.

3. Data

  • The project defines three dataset formats: audio-transcription pairs for HuBERT, discrete-unit-audio pairs for the vocoder, and audio-text-unit triplets for the SLM.The system also requires a Whisper encoder, a HiFi-GAN vocoder, a HuBERT unit encoder, and an LLM.Source: §3 · p. 2

3.1. HuBERT training Dataset

  • 600 hours of transcribed Basque audio from Common-Voice [12], BasqueParliament, and OpenSLR were curated for ASR fine-tuning of the HuBERT encoder.The corpus is the composite-eu dataset.Source: §3.1 · p. 2

3.2. Vocoder Dataset

  • The Basque TTS dataset provides two speakers, each with 13,500 recorded utterances averaging four seconds, or about 16 hours of speech per speaker.These audio and discrete-unit pairs are used to train the vocoder.Source: §3.2 · p. 2

3.3. SLM Dataset

  • About 450,000 translated Basque questions yielded 550 hours of questions and over 3,300 hours of answers through 14-voice synthesis and discrete-unit extraction.VoiceAssistant-400K [7] was translated with Latxa-3.1-70B-Instruct [8]; target responses were synthesized using the vocoder's speaker, and HuBERT-derived units were extracted for stage-two training.Source: §3.3 · p. 3

4. Training

Training adapts the HuBERT encoder and vocoder before training the SLM in two stages. The stages first teach speech understanding and text generation, then speech-response generation.

4. Training

  • The overall training sequence adapts the discrete-token speech encoder and vocoder before the complete SLM undergoes two-stage training.The stages are speech understanding and text generation, followed by speech response generation.Source: §4 · p. 3, §4.3 · p. 3

4.1. HuBERT Training

  • Because continuous HuBERT pretraining requires more Basque data than is available, the authors fine-tuned the encoder for ASR using transcribed speech.They started from the multilingual mHuBERT-147 checkpoint, which includes Basque.Source: §4.1 · p. 3

4.2. Vocoder Training

  • The Unit HiFi-GAN vocoder uses speaker embeddings from ECAPA-TDNN; early experiments favored a two-speaker setup for acoustic quality and stability without broader-configuration complexity.The final vocoder was trained on Maider and Antton.Source: §4.2 · p. 3

4.3. SLM Training

  • In stage 1, the encoder stays frozen while the adaptor and LLM learn concise text responses; in stage 2, only the streaming decoder learns speech units from LLM hidden states using CTC loss.The asynchronous CTC alignment handles variable sequence lengths without explicit text-to-speech alignment; each stage ran for three epochs on four A100 GPUs in under 100 hours total.Source: §4.3 · p. 3

5. Evaluation and Results

The evaluation covers the HuBERT encoder and Unit HiFi-GAN vocoder separately, then assesses the complete dialogue system on content quality, audio quality, and intelligibility. The end-to-end system scores 3.86/5 for content, 3.07 UTMOS, and 9.94% WER, with cross-language comparisons described as indicative rather than strictly comparable.

5. Evaluation and Results

  • The evaluation separates the HuBERT discrete-unit encoder and Unit HiFi-GAN vocoder from the complete end-to-end dialogue system.For the end-to-end system, content, audio quality, and speech intelligibility are evaluated; cross-language scores use different judges and ASR systems, so they are indicative rather than strictly comparable.Source: §5 · p. 3, §5.2 · p. 4, Table 1 · p. 4

5.1. Modules

  • 4.90% WER on the composite-eu test split followed 100 HuBERT training epochs, with representations reported to approach state-of-the-art Basque ASR baselines.The encoder was fine-tuned on an ASR task to evaluate its learned representations.Source: §5.1 · p. 4
  • After 1000 training epochs with two speakers, Unit HiFi-GAN achieved 4.8 dB MCD for Antton and 5.1 dB for Maider.MCD measures vocoder waveform synthesis from discrete tokens; values under 5–6 dB typically indicate high synthesis quality.Source: §5.1 · p. 4

5.2. End-to-End SLM

  • 3.86/5 LLM-as-a-judge, 3.07 UTMOS, 9.94% WER, and 1.62% CER are the end-to-end system’s reported scores.Scores were computed over 4,419 synthetic instructions disjoint from training data; the evaluation is in-distribution, and real-user speech remains untested.Source: §5.2 · p. 4
  • The Basque model’s 3.86 content score is close to LLaMA-Omni’s 3.99 and above Qwen2-Audio’s 3.47 and SALMONN’s 3.44; its 9.94% WER is comparable to LLaMA-Omni’s 10.82%.The 3.07 UTMOS score falls behind all English baselines; streaming latency is 370 ms, versus 346 ms for LLaMA-Omni and 5587 ms for SpeechGPT.Source: §5.2 · p. 4, Table 1 · p. 4

6. Conclusions

The paper presents an open-source end-to-end Basque SLM and a reproducible modular framework that uses synthetic spoken-instruction data rather than massive foundational pre-training. Its results support transferring Basque language capabilities to speech, while real nonsynthetic speech and improved acoustic quality remain future-work targets.

6. Conclusions

  • The first open-source end-to-end Basque SLM establishes a reproducible framework for low-resource language speech interaction without massive foundational pre-training.The authors frame it as a way to bridge the multimodal gap for low-resource languages.Source: §6 · p. 4
  • In the absence of spoken-instruction data, the authors report automated synthetic data generation as effective, with 3.86/5 content quality and competitiveness with established English-centric systems.Their workflow translates English corpora with LLMs and uses multi-speaker TTS for multimodal alignment.Source: §6 · p. 4
  • Specialized speech adaptors and parallel streaming decoders transfer a language-specific text backbone’s linguistic capabilities to speech, with 9.94% WER and 370 ms streaming latency reported.The WER is described as comparable to original English LLaMA-Omni across different languages and ASR systems.Source: §6 · p. 4
  • Future work targets more expressive TTS, a second judge and small-scale human evaluation, and testing on real nonsynthetic Basque speech.These are identified as next steps for addressing the acoustic-quality gap and supplementing the automatic evaluation.Source: §6 · p. 4

Related papers

Loading 2610.36911v1…