PaperlayerView exact revision on arXiv

Source-linked AI summary

Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs

Sara Papi, Javier Garcia Gilabert, Zachary Hopton, Vilém Zouhar, Carlos Escolano, Gerard I. Gállego, Jorge Iranzo-Sánchez, Ahrii Kim, Dominik Macháček, Patricia Schmidtova, Maike Züfle

arXiv:2512.16378v4cs.CLcs.AIcs.SD

TL;DR

Whether integrating speech into LLMs improves speech-translation quality over established direct and cascaded systems remains unresolved. Hearing to Translate benchmarks SpeechLLMs and other ST systems across languages, datasets, and challenging conditions. Cascades remain most reliable overall, although recent SpeechLLMs match or outperform them in some settings and standalone SFMs lag behind.

  • Problem

    Whether SpeechLLMs can match or surpass translation-specialized direct and cascaded systems remains unclear, and prior work rarely compares the paradigms systematically across challenging speech conditions.

    Source: §1 · p. 2

  • Method

    Hearing to Translate benchmarks 6 state-of-the-art SpeechLLMs against 16 direct and cascaded systems across 13 language pairs, 16 benchmarks, and 9 conditions.

    Source: §1 · p. 2

  • Results

    Cascaded systems remain most reliable overall, while recent SpeechLLMs match or outperform them in some settings and standalone SFMs lag behind both.

    Source: §7 · pp. 12–13

  • Takeaways & Limitations

    The results highlight the importance of an LLM, integrated or used in a cascade, for high-quality speech translation.

    Source: §7 · p. 13

    Long-form evaluation excludes Seamless and Spire because they lack native long-form inference support.

    Source: DISFLUENCIES NAMED ENTITIES NOISE… · p. 8

Abstract

from arXiv · show

As Large Language Models (LLMs) expand beyond text, integrating speech as a native modality has given rise to SpeechLLMs, which directly process spoken language and enable speech-to-text translation (ST) and other downstream tasks, bypassing traditional transcription-based pipelines. Whether this integration improves ST quality over established cascaded architectures, however, remains an open question. We present Hearing to Translate, the first comprehensive test suite rigorously benchmarking 6 state-of-the-art SpeechLLMs against 16 strong direct and cascade systems that couple leading speech foundation models (SFM), with multilingual LLMs. Our analysis spans 16 benchmarks, 13 language pairs, and 9 challenging conditions, including disfluent, noisy, and long-form speech. Across this extensive evaluation, we find that cascaded systems remain the most reliable solution overall, but most recent SpeechLLMs can match or even outperform cascades in various settings while SFMs lag behind both, highlighting that integrating an LLM, either within the model or in a pipeline, is essential for high-quality speech translation.

1 Introduction

SpeechLLMs bring spoken language into LLMs, but whether this integration improves speech translation over strong direct and cascaded systems remains unresolved. Hearing to Translate addresses that question with a broad comparison across systems, languages, benchmarks, and challenging conditions.

Motivation

  • Speech conveys linguistic content as well as prosodic, emotional, and paralinguistic cues, motivating its integration into multimodal LLMs.The paper frames speech as a central, information-rich form of human communication.Source: §1 · p. 1

ST paradigms

  • Cascaded ST transcribes speech before translation, while direct ST maps speech to translated text; direct models can be data-hungry and less flexible at test time.Cascades can be fine-tuned component by component but are vulnerable to ASR error propagation, added latency and computation, and loss of prosodic or paralinguistic cues.Source: §1 · p. 2

Research gap

  • Whether SpeechLLMs match or surpass translation-specialized direct or cascaded systems remains an open question, and prior comparisons rarely address real-world speech phenomena systematically.The paper highlights disfluencies, background noise, and code-switching as examples.Source: §1 · p. 2

SpeechLLMs

  • SpeechLLMs combine speech processing with LLM linguistic knowledge and contextual flexibility, but their practical translation benefits remain uncertain.A typical SpeechLLM connects an audio encoder to an LLM through adapters bridging acoustic and LLM representations.Source: §1 · pp. 1–2

Study scope

  • The study compares 6 state-of-the-art SpeechLLMs with 16 strong direct and cascaded systems across 13 language pairs, 16 benchmarks, and 9 conditions.The conditions cover varied linguistic and acoustic phenomena in realistic settings.Source: §1 · p. 2

2 Related Works

Prior work has compared cascaded and direct speech translation and established LLM strengths in text translation, but systematic comparisons of LLM-enhanced speech translation remain limited. This study revisits the balance among these approaches in the SpeechLLM era.

Cascaded vs. Direct ST: A Historical Comparison

  • Recent IWSLT campaigns report cascades combining strong SFMs and high-performing LLMs outperforming direct ST across multiple language pairs and acoustic conditions.Earlier work had highlighted potential end-to-end benefits in reducing error propagation and latency while achieving comparable or better results.Source: §2 · p. 2
  • Systematic comparisons of SpeechLLMs with other ST paradigms remain limited, leaving their place in the established balance between cascaded and direct systems unsettled.Prior studies also report difficulty generalizing in realistic multilingual or low-resource settings for direct systems.Source: §2 · p. 2

The LLM Era is Here, for MT

  • LLMs’ text-translation strengths and contextual abilities have motivated SpeechLLMs, but whether those advantages extend to speech translation had not been verified.Reported text-side capabilities include multilingual coverage, contextual reasoning, in-context learning, and adaptation to user intent, style, and formality.Source: §2 · pp. 2–3

3 The Hearing-to-Translate Suite

The Hearing-to-Translate Suite organizes evaluation around nine challenging speech conditions, benchmarks selected or adapted to cover them, and metrics for translation quality and robustness. Its design combines quality-estimation measures with task-specific accuracy and condition-gap measures.

3.1 Categorization of Analyzed Phenomena

  • The suite evaluates nine linguistic, acoustic, and sociolinguistic conditions, from clean speech and gender bias to code-switching, noise, emotion, and long-form inputs.It also includes accents, disfluencies, and named entities; long-form examples may span several minutes and test contextual consistency and memory handling.Source: §3.1 · p. 3

3.2 Benchmarks

  • Benchmarks are selected or created to cover the suite’s conditions, including NoisyFLEURS with babble and ambient noise and LibriStutter with automatically inserted disfluencies.The set also includes datasets for accent and dialect variation, code-switching, named entities, emotion, long-form speech, and crosslingual instruction-following.Source: §3.2 · p. 3, GENERIC · pp. 4–5
  • Table 1 summarizes benchmarks, covered phenomena, and source languages; FLEURS supports gender-bias analysis through balanced speaker sex where possible.The benchmark descriptions also note that WMT speech-domain segments contain background noise and include human reference translations.Source: §3.2 · p. 3, Table 1 · p. 4

3.3 Metrics

  • Because speech benchmarks often lack reference translations, the evaluation relies on QE metrics, including modified xCOMET and METRICX versions that penalize off-target outputs.The suite also reports exact-match accuracy for named entities and terminology; %NE is defined from exact named-entity matches against reference entities.Source: §3.3 · pp. 5–6
  • Performance gaps compare scores on contrasting subsets: near-zero values indicate comparable performance, while positive or negative values favor subset A or B, respectively.The defined comparisons include accent, disfluency, noise, long-form length, and speaker-gender conditions; WinoST also uses a coreference F1 gap.Source: §3.3 · p. 5

4 Experimental Settings

The experimental setup selects models across speech-foundation, cascade, LLM, and SpeechLLM paradigms, then evaluates supported language directions with translation prompts or model-specific language instructions. Inference uses default decoding settings to reflect out-of-the-box use, with a stated exception for Spire.

4.1 Models

  • The comparison includes SFMs, SFM-based cascades, LLMs, and six SpeechLLMs, with models limited to fewer than 32 billion parameters.The selected SpeechLLMs are Phi-4-Multimodal, Qwen2-Audio, Qwen3-Omni, DeSTA2, Voxtral, and Spire.Source: §4.1 · p. 6

4.2 Languages and Inference

  • Evaluation covers six source-to-English and seven English-to-target directions, using common supported languages across the analyzed SpeechLLMs.LLMs use an adapted WMT 2025 General MT translation prompt; SFMs receive target-language or source-and-target-language instructions, depending on the model.Source: §4.2 · p. 6
  • Default decoding settings are used for out-of-the-box performance, except Spire, which uses beam search with beam size 5 after default settings produced unusable outputs.Most inference runs use Hugging Face Transformers; OWSM and Canary use ESPnet and NVIDIA NeMo, respectively.Source: §4.2 · p. 6

5 Results

Cascaded systems remain strongest overall, but no paradigm dominates every condition: the best SpeechLLMs match or surpass cascades in some settings, while standalone SFMs generally lag. Human evaluation broadly mirrors automated results, though automatic metrics distinguish systems more reliably than individual outputs.

GENERIC GENDER BIAS ACCENTS CODE SWITCHING

  • Gender disparities depend substantially on the text-generation module, and gender coreference and stereotype gaps correlate (r = 0.54); named-entity accuracy does not perfectly track overall translation quality.Pairing ASR with Gemma3 yields substantial gender gaps, while Tower+ mitigates them; LLM choice also plays a central role in named-entity and terminology accuracy.Source: §5.1 · p. 6, DISFLUENCIES NAMED ENTITIES NOISE… · p. 8, §5.2 · p. 9

5.1 Overall Results

  • Cascades are difficult to outperform overall, while Voxtral and Qwen3-Omni reliably close or sometimes overturn the gap; standalone SFMs generally lag.The strongest average results come from Canary and Whisper paired with Aya, Qwen3-Omni, and Voxtral.Source: §5.1 · p. 6, Table 2 · p. 7

DISFLUENCIES NAMED ENTITIES NOISE EMOTION LONG-FORM

  • Voxtral, DeSTA2, and Whisper cascades are most robust to stuttering; cascaded Whisper and especially Voxtral lead on code switching.The code-switching results indicate that both encoder and decoder matter; disfluency robustness is not driven solely by the speech encoder.Source: DISFLUENCIES NAMED ENTITIES NOISE… · p. 8
  • Under noise, most SpeechLLMs are at least as robust as SFMs and cascades, while cascades are more robust on emotion except for Qwen3-Omni.All systems degrade under noise, especially babble noise; the reported minimum is 38∆noise.Source: DISFLUENCIES NAMED ENTITIES NOISE… · p. 8
  • DeSTA2 and Qwen2-Audio show ∆length≈91–94, whereas OWSM, Canary, and Voxtral cascades have near-zero ∆length; Voxtral is the strongest long-form SpeechLLM.Voxtral re-concatenates chunk representations before the LLM, allowing long-context speech translation.Source: DISFLUENCIES NAMED ENTITIES NOISE… · p. 8

5.2 Analysis

  • Strong performance on standard accents does not reliably transfer: several SpeechLLMs are unstable on CommonAccent, while cascades show large standard-Mandarin versus dialect gaps on ManDi.Across accents, x-en generally underperforms en-x, with Phi-4-Multimodal a notable exception (xCOMETQE S 80.5 vs. 75.1).Source: DISFLUENCIES NAMED ENTITIES NOISE… · p. 8, §5.2 · p. 9

5.3 Human Evaluation

  • Human evaluation broadly matches automated results: Canary+Aya consistently outperforms Voxtral and Seamless on x-en, while SpeechLLM omissions are twice as frequent as in cascades and SFMs.Wrong terminology accounts for 11.5–12.5% of identified errors.Source: Table 3 · p. 10, §5.3 · p. 10, Table 4 · p. 10
  • Automatic metrics distinguish systems more reliably than individual outputs: global human-score correlations are around 0.5 across most languages, but item-level ranking remains difficult.The reported global correlation is micro-Pearson; item-level ranking uses group-by-item Spearman.Source: Table 5 · p. 10, §5.3 · p. 10

6 Discussion

The discussion compares proprietary and open-weight systems, isolates the effect of a shared LLM backbone, and weighs translation quality against computational cost. It also examines training mixtures, prompt consistency, and the validity of the evaluation metrics.

6 Discussion

  • Gemini-2.5-flash is competitive on GENERIC, while top open-weight systems match or surpass it overall and differ by task on gender gaps and named entities.On the GENDER BIAS FLEURS subset, Gemini shows small gender gaps comparable to the best open-weight systems; on NAMED ENTITIES, it beats the best SFM and cascade on NE and terminology accuracy, but trails the best SpeechLLM on NE.Source: §6 · pp. 10–11, Table 6 · p. 11
  • Aya cascades and Qwen3-Omni require ×27–34 the latency and ×6–14 the memory of benchmarked SFMs, illustrating a translation-quality and efficiency trade-off.Voxtral has up to ×33 lower latency than Aya-based cascades and Qwen3-Omni, while requiring up to +52GB more memory than SFMs and Tower+-based cascades.Source: Figure 3 · p. 11, §6 · p. 11
  • Among SpeechLLMs, Voxtral and Qwen3-Omni, which include speech during pretraining, tend to outperform systems that add speech only during adaptation.The authors suggest early multimodal alignment may play an important role in integrating speech into LLMs effectively.Source: §6 · pp. 11–12
  • With a shared Qwen3 backbone, Qwen3-Omni generally outperforms cascades, though cascades win slightly in a few cases; the gains require additional speech-encoder integration training.The comparison is designed to partially isolate speech-modality integration from differences in LLM training, quality, and robustness.Source: Table 7 · p. 12, §6 · p. 12
  • All models use the same prompt template and suggested decoding configuration; prompt reformulations may change absolute scores but are unlikely to substantially alter comparative conclusions.The standardized setup is intended to make performance differences reflect off-the-shelf model use.Source: §6 · p. 12
  • Quality-estimation metrics closely track reference-based metrics, with global correlations approaching 1.0, and METRICX reaches 0.57 system-level correlation with human judgments.The human evaluation was limited in scale, but the metric-to-metric and metric-to-human consistency supports the reported architectural rankings.Source: §6 · p. 12, Table 8 · p. 12

7 Conclusions

Across 22 systems, 13 language pairs, nine phenomena, and 16 benchmarks, cascades remain the most reliable overall, while recent SpeechLLMs can match or outperform them in some conditions. The conclusions also identify persistent gender, accent, and translation-error challenges.

7 Conclusions

  • Cascades remain the most reliable overall, but recent SpeechLLMs can match or outperform them in noise, code-switching, and disfluencies, while standalone SFMs lag behind both.The authors conclude that an LLM, integrated into the model or used in a cascade, is crucial for high-quality speech translation.Source: §7 · p. 13
  • All three paradigms struggle to use contextual cues for gender assignment and often default to masculine forms; accent sensitivity causes substantial performance variation, particularly for SpeechLLMs.The paper attributes most gender bias to the LLM component.Source: §7 · p. 13
  • Mistranslations, terminology errors, and overtranslation are recurring human-evaluation findings, with overtranslation especially prevalent in LLM-based systems.Alignment with automatic metrics supports the evaluation framework.Source: §7 · p. 13

A Human Evaluation Interface

The paper directs readers to Figure 4 for the human-evaluation annotation guidelines and interface.

A Human Evaluation Interface

B Evaluation Settings

This appendix records evaluation settings and the model and prompt choices used in the study. It documents selected speech foundation models, multilingual LLMs, and SpeechLLMs, along with their relevant configurations and scope.

B Evaluation Settings

  • The evaluation settings identify Python 3.9.16 and specify the xCOMET and METRICX model variants and scoring libraries used.xCOMET scores use unbabel/xcomet-xxl with comet 2.2.2 in fp32; METRICX uses google/metricx-24-hybrid-xxl-v2p6-bfloat16.Source: B Evaluation Settings · p. 23

C Model Details

C.1 SFMs

  • The SFM set includes Whisper large-v3, SeamlessM4T v2-large, Canary v2, and OWSM 4.0; Whisper uses 30-second chunked decoding, while Seamless is not evaluated on long-form speech.OWSM uses a 1B-parameter CTC encoder-only variant selected for long-form robustness and faster inference.Source: §C.1 · pp. 23–24

C.2 LLMs

  • The LLM selection includes Tower+ 9B, Gemma 3 12B, and Aya Expanse 32B, chosen to represent translation-specific, small, and medium-sized categories.Aya was selected over Gemma 3 27B for stronger performance across more language pairs and model diversity.Source: §C.2 · p. 24

C.3 SpeechLLMs

  • The SpeechLLM set includes models with different speech interfaces and training designs: Spire uses discretized speech units, DeSTA2 combines Whisper transcripts and speech features, and Voxtral uses a Whisper encoder with an LLM decoder.Spire is limited to speech recognition and translation under tightly defined instructions and has no general-purpose capabilities; the study also describes Phi-4-Multimodal, Qwen2-Audio, and Qwen3-Omni.Source: §C.3 · pp. 25–26

D Prompts

  • The LLM and SpeechLLM prompts use translator instructions with source and target language placeholders, while Spire uses its trained prompt template.Source: D Prompts · p. 26

E Limitations

The evaluation has three stated boundaries: its English-centric model coverage, its exclusion of traditional neural MT, and the absence of toxicity and safety benchmarks.

E Limitations

  • The analysis is English-centric because available SpeechLLMs currently limit language support; a fully multilingual setup would require broader model coverage and additional resources.The authors also note that they omit traditional neural MT and toxicity or safety benchmarks; no public datasets currently target the latter in speech-to-text translation.Source: E Limitations · p. 27
  • Traditional neural MT results are not reported because the study focuses on speech integration in LLMs and direct or cascaded speech-to-text translation pipelines.Source: E Limitations · p. 27
  • Toxicity and safety benchmarks are excluded because no publicly available datasets currently target these aspects in speech-to-text translation.Source: E Limitations · p. 27

F METRICXQE

Overall results use METRICXQES transformed to a [0, 100] scale for consistency with xCOMETQES, where higher values indicate better performance.

F METRICXQE

  • The paper reports overall results using METRICXQES.Source: S Overall Results · p. 28
  • METRICXQES scores are transformed as 100 − 4 · METRICXQES to map them to [0, 100] and align with xCOMETQES, where higher values indicate better performance.Source: S Overall Results · p. 28

G Accent-specific Results

Accent-specific results are shown by source accent, with the reported language-pair coverage and score aggregation conventions varying by direction.

GENERIC GENDER BIAS ACCENTS CODE SWITCHING

G Accent-specific Results

  • Figures 5 and 6 break down xCOMETQES by source accent: Figure 5 covers pairs into English, using ManDi for zh-en and CommonAccent for other pairs, while Figure 6 covers pairs out of English on CommonAccent.Source: Figure 5 · p. 29, Figure 6 · p. 30

DISFLUENCIES NOISE EMOTION LONG-FORM

Related papers

Loading 2512.16378v4…