SeamlessM4T: Massively Multilingual & Multimodal Machine Translation

• Speech translation systems lag text systems in scope and performance, often focusing on high-resource languages and translation into English, while cascades remain common and unified models underperform them.The introduction identifies audio-data scarcity and modeling constraints as obstacles despite demand for on-demand speech-to-speech translation.Source: §1 · p. 4
• The system combines representations learned from 1 million hours of open speech with SeamlessAlign, a 470,000+ hour corpus, and 406,000 hours of filtered, human-labeled, and pseudo-labeled data.It supports S2ST from 100 languages to English and English to 35; S2TT in 100-to-English and English-to-95 directions; ASR in 96 languages; zero-shot T2ST in 95-to-English and English-to-35 directions; and T2TT in 95-to-English and English-to-95 directions.Source: §1 · pp. 4–5
• SeamlessM4T-Large beats AudioPaLM-2-8B-AST by 4.2 BLEU points (20%) on Fleurs [Conneau et al., 2022] for translation into English, and cascaded models by over 2 BLEU points.Source: §1 · p. 5
• On Fleurs, SeamlessM4T-Large exceeds three-stage cascaded S2ST models by 2.6 ASR-BLEU points and a two-stage model by 8.5 ASR-BLEU points (50%) on CVSS.The two-stage baseline is Whisper-Large-v2 + YourTTS.Source: §1 · p. 5
• It improves over XLS-R-2B-S2T by 2.8 BLEU points on CoVoST 2 [Wang et al., 2021c] and reduces Fleurs ASR WER by 45% versus Whisper-Large-v2 [Radford et al., 2022].On Flores, it matches NLLB-3.3B [NLLB Team et al., 2022] into English and averages a 1 chrF++ point improvement from English; robustness improves by 38% against background noise and 49% against speaker variation. Human evaluations found XSTS scores above 4/5 for 24 English-source languages and significant gains over Whisper-Large-v2 in 7 of 24 into-English languages.Source: §1 · p. 5
• Added toxicity prevalence is 0.11%–0.21%, with reductions of 26%–63% versus state-of-the-art models; the system overgeneralizes masculine forms by about 10% and shows about 3% gender-variation robustness gaps.Gender-bias results were comparable to state-of-the-art models, motivating further mitigation.Source: §1 · p. 6
• The work releases SeamlessM4T models, inference and fine-tuning tools, speech-data alignment resources, and the text-free S2ST evaluation model Blaser 2.0.Source: §1 · p. 7

2.1 Why Prioritize Speech in Machine Translation?

• Speech carries expressive and social cues that text lacks, and spoken exchanges can strengthen bonds and perceived warmth [Plag et al., 1999; Kraut et al., 1992; Kumar and Epley, 2021, 595; Schroeder et al., 2017].Speech can differ from text in grammar, registers, and morphology, while volume, intonation, and pace help convey human traits.Source: §2.1 · pp. 7–8
• Speech translation can improve access for people underserved by text, and speech outputs can bypass barriers from unfamiliar or changing writing systems.The section cites 773 million adults without the education needed to read or write, and 43 million blind people plus 295 million with moderate to severe visual impairment [Markelova, 2021; GBD 2019 Blindness and Vision Impairment Collaborators, 2021]. Urdu may use Arabic or Devanagari scripts, while Uzbek has changed writing systems five times; speech tools may stabilize access during transitions.Source: §2.1 · p. 8

2.2 Speech Translation Today

• Cascades can reuse advances in ASR, translation, and synthesis, but transcription errors, error propagation, domain mismatch, and loss of paralinguistic cues can reduce translation quality [NLLB Team et al., 2022; Radford et al., 2022].A two-stage ASR–T2TT system can fall short of a single large T2TT model; proper names and nouns may also be mishandled.Source: §2.2 · p. 9
• End-to-end S2TT has reached parity with cascades in constrained settings, while direct S2ST progressed from Translatotron to two-pass decoding and speech-unit approaches [Ansari et al., 2020; Potapczyk and Przybysz, 2020b; Jia et al., 2019b; Jia et al., 2022a; Tjandra et al., 2019; Lee et al., 2022a].Translatotron-2 uses intermediate phoneme decoding; other approaches translate between discrete speech units and use a speech re-synthesizer.Source: §2.2 · pp. 9–10
• Despite multilingual scale, USM, Whisper, and AudioPaLM primarily support or report translation into English, leaving translation from English and broader target coverage as gaps [Radford et al., 2022].AudioPaLM is described as the current SOTA for S2TT and S2ST, but its reported translation outputs are into English.Source: §2.2 · p. 10
• The effort aims to unify S2ST, S2TT, T2ST, T2TT, and ASR, expand supported languages and directions, and evaluate safety and robustness systematically.The stated goals include bridging gaps between high- and low-resource languages and improving equity and fairness across user demographics.Source: §2.2 · pp. 11–12

2.3 Languages

• Because existing speech translation models cover 21 to 113 source languages but lack T2TT, the authors set a 100-language speech-source goal for a unified speech-and-text model.The language inventory records resource levels and whether each language is supported as a speech or text source or target.Source: §2.3 · p. 12, Table 5 · p. 15

3.1 Speech-language identification

• The custom LID model addresses gaps in available language coverage; incorrect audio labels can block high-quality alignments or add noise to paired data.The model follows ECAPA-TDNN and adds support for languages including Moroccan and Egyptian Arabic, Central Kurdish, Irish, Igbo, and Zulu.Source: §3.1 · pp. 16–17, §3 · p. 16

3.1.1 Training

• 14.6% average gain across 17 languages accompanied the LID model’s F1 improvement greater than 5%, while 12 languages averaged a 9.8% loss.Filtering rejected 8% of data and increased F1 by almost 3%.Source: §3.1.1 · pp. 17–18, Table 6 · p. 18, §3.1.2 · p. 18

3.2 Gathering raw audio and text data at scale

• Four million hours of raw audio were curated and over-segmented into overlapping candidate splits, roughly octupling the segments considered for mining.Over-segmentation leaves the mining algorithm to choose optimal alignments because speech pauses do not reliably mark sentence boundaries.Source: §3 · p. 16, §3.2 · pp. 18–19

3.3.1 Sonar text embedding space

• Sonar substantially outperforms Laser3 and LaBSE on multilingual similarity-search error rates, a proxy for text-to-text mining.Its fixed-size representation bottleneck decreases BLEU by 7% into English (35.2→32.7) and 13% out of English (24.9→21.6).Source: §3.3.1 · pp. 20–21, Table 9 · p. 21

3.3.2 Training speech encoders

• Sonar speech encoders use teacher-student training with MSE loss, a w2v-BERT 2.0 front end, attention pooling, and human-performed ASR transcriptions.Languages are grouped by linguistic family; the resulting speech embeddings can be decoded with the Sonar text decoder.Source: §3.3.2 · p. 21
• Zero-shot Sonar speech translation outperforms Whisper-Large-v2 on several low-resource languages, particularly Swahili, Bengali, Kannada, Telugu, and Tamil, despite gaps on some high-resource languages.Evaluation decodes foreign speech embeddings into English; the encoders were trained only on ASR transcriptions, and the text decoder had never received speech input.Source: Table 10 · p. 22, §3.3.2 · p. 23

3.3.3 Speech mining

• Mining uses global nearest-neighbor comparisons and a Sonar margin threshold of 1.15; English-speech-to-foreign-text mining was limited to some languages, and full mined metadata and code are open-sourced.The threshold was set after human inspection because end-to-end evaluation was too compute-intensive; other mining directions remain future work.Source: §3.3.3 · pp. 23–24
• 470,000 hours across 37 languages comprise approximately 200,000 hours English speech to non-English text (Sen2Txx), 240,000 hours non-English speech to English text (Sxx2Ten), and 29,000 hours non-English speech to English speech (Sxx2Sen).More than 100 hours of English-speech alignments were mined for all languages except Maltese; alignments with English text exceeded 10,000 hours for six languages.Source: Table 10 · p. 22, §3.3.3 · p. 24

3.4.2 Speech segmentation

• VAD pause-based segmentation was augmented with Duquenne et al.’s [2021] over-segmentation strategy, which increases mining recall but creates 8x more segments, noise, and computational demand.The passage notes that pause-based segments may be too short or semantically incoherent, and reviews alternatives including SHAS, which uses wav2vec 2.0 splitting probabilities with a divide-and-conquer algorithm.Source: §3.4.2 · p. 25

3.4.3 Multilingual and multimodal representations

• Sonar is presented as the first sentence-embedding space with an efficient multilingual decoder, alongside multilingual text and speech representation approaches.Prior work includes Laser, LaBSE, and teacher-student extensions such as Laser3, which extended Laser to 200 languages; speech research includes unsupervised and joint text-speech models, and fixed-size representations for multimodal mining.Source: §3.4.3 · pp. 25–26

3.4.4 Speech mining

• SpeechMatrix aligned speech across 17 languages in 136 language combinations, totaling 418,000 hours, including about 46,000 hours aligned with English.Its audio comes from VoxPopuli parliamentary speeches, limiting its domain; speech synthesis on text-aligned corpora is another route to speech-to-speech data, as in CVSS [Jia et al., 2022b].Source: §3.4.4 · p. 26

4. SeamlessM4T Models

• SeamlessM4T aims to bridge the direct-versus-cascaded S2TT gap with X2T and enables S2ST through UnitY, which generates text before discrete acoustic units and can be jointly optimized.Its four building blocks are SeamlessM4T-NLLB, w2v-BERT 2.0, T2U, and a multilingual HiFi-GAN unit vocoder.Source: §4 · p. 27, Figure 4 · p. 28

4.1 Unsupervised Speech Pre-training

• w2v-BERT 2.0 combines contrastive learning with masked prediction, using two GVQ codebooks and an additional masked-prediction task with RPQ codes.The pre-trained encoder has about 600M parameters and is trained on 1 million hours of open speech audio covering over 143 languages.Source: §4.1 · pp. 28–29

4.2 X2T: Into-Text Translation and Transcription

• X2T joins a Conformer speech encoder and a Transformer text encoder to one text decoder, and trains on speech paired with target-language text.Source: §4.2 · pp. 29–30

4.2.1 Preparing X2T data

• To expand scarce S2TT supervision, the authors pseudo-label ASR data with NLLB-200-3.3B and mine 18.3K hours of speech from selected SeamlessAlign directions.Mining selects the top 400 hours in each of 33 X–eng directions and 200 hours in each of 29 eng–X directions by Sonar alignment scores.Source: §4.2.1 · p. 30
• 84% of the MTSU top 5K Chinese characters are covered by the retrained 256K SentencePiece vocabulary, up from 54%.Source: Filtering · p. 31

4.2.2 Training a Large-Scale Multilingual Text-to-Text Translation Model

• SeamlessM4T-NLLB outperforms the 1.3B and 1.3B-distil NLLB-200 models and is on par with the 3.3B model on average Flores devtest chrF++ across 95 languages.The comparison covers translations from English and into English.Source: §4.2.2 · p. 32, Table 12 · p. 32

4.2.3 Multimodal & multitask into target text

• Joint X2T fine-tuning combines speech and text encoders with a shared text decoder and optimizes S2TT, T2TT, and token-level knowledge distillation losses across two training stages.Stage 1 uses supervised English ASR and into-English S2TT; Stage 2 adds English-to-X S2TT and non-English ASR. When data do not form triplets, translation objectives are replaced with auto-encoding.Source: §4.2.3 · pp. 32–33

4.3 Speech-to-Speech Translation

• UnitY decomposes S2ST into speech-to-unit translation and unit-to-speech conversion, generating text first and discrete acoustic units second.Its S2UT model replaces the scratch-trained core with a pretrained X2T model and uses a deeper, pretrained T2U component.Source: §4.3 · pp. 33–34

4.3.1 Preparing S2ST data

• K=10000 centroids from 35th-layer XLS-R-1B features gave the best speech re-synthesis WER among tested codebook sizes.XLS-R had broader language coverage than HuBERT and similar re-synthesis performance on overlapping languages; w2v-BERT 2.0 performed worse.Source: §4.3.1 · p. 34
• A multilingual HiFi-GAN vocoder converted predicted units into waveforms for 36 target languages, using language embeddings and an auxiliary language-identification loss.Its duration-predictor and speech-language-identification-classifier embedding dimensions were doubled to 1280 relative to monolingual vocoders.Source: §4.3.1 · p. 34
• To address scarce parallel S2ST data, multilingual text-to-unit models generated units directly from text, avoiding slow TTS-based synthesis across 36 target languages.Cross-lingual transfer supported low-resource languages, and pairs with more than 0.5 seconds of generated audio per text token were discarded.Source: §4.3.1 · pp. 34–35
• SeamlessAlign supplied up to 2,500 hours of mined S2ST data per language direction, with target units extracted by the XLS-R pipeline.An in-house ASR model generated target-speech transcriptions for the first-pass decoder.Source: §4.3.1 · p. 35

4.3.2 T2U modeling

• T2U models served pseudo-labeling and UnitY initialization: the former used 12-layer encoders and decoders, while the latter used six-layer encoders and decoders.Fine-tuning the smaller model on labels from the larger one bridged the quality gap while remaining parameter-efficient.Source: §4.3.2 · pp. 35–36

4.3.3 Stage3 Finetuning for S2ST

• Stage3 initialized UnitY from pretrained X2T and T2U, then fine-tuned only T2U on 121K hours of X–eng and eng–X S2ST data.Freezing X2T was intended to preserve performance on tasks from earlier fine-tuning stages.Source: Figure 7 · p. 33, §4.3.3 · p. 36

4.4 The SeamlessM4T Models

• SeamlessM4T-Large has 2.3B parameters and is fine-tuned across T2TT, ASR, S2TT, and S2ST, with coverage ranging from 89 to 96 languages for the first three tasks and 95 directions into English plus 35 target languages out of English for S2ST.SeamlessM4T-Medium has 57% fewer parameters than Large while retaining the same coverage, using a 300M-parameter w2v-BERT 2.0 and NLLB-600M-Distilled initialization.Source: §4.4 · p. 36

4.4.1 Comparison to cascaded approaches.

• Against cascaded S2TT systems with fewer than 3B parameters, SeamlessM4T-Large gains 2 BLEU points in X–eng and 0.5 in eng–X; versus Whisper-Large-v2 + NLLB-3.3B, it gains 1.3 BLEU points on average in X–eng.Cascades using NLLB-3.3B exceed 4B parameters and outperform SeamlessM4T-Large only in eng–X directions.Source: §4.4.1 · p. 37, Table 14 · p. 38
• SeamlessM4T-Large beats cascaded S2ST systems by 9 ASR-BLEU points on Fleurs X–eng versus 2-stage systems and by 2.6 points versus a stronger 3-stage system.On CVSS X–eng, it beats the 2-stage Whisper-Large-v2 + YourTTS system by 14 ASR-BLEU points.Source: §4.4.1 · p. 37, Table 15 · p. 38

4.4.2 Multitasking X2T results.

• On Fleurs S2TT X–eng, SeamlessM4T exceeds AudioPaLM-2 8B AST by 4.2 BLEU points (20%); on CoVoST 2 eng–X, it beats XLS-R by 2.8 BLEU points.It trails AudioPaLM by 3.7 BLEU points on CoVoST 2 X–eng; on Fleurs, its ASR yields a 45% WER reduction over Whisper [Radford et al., 2022] across 77 overlapping languages.Source: §4.4.2 · pp. 37–38, Table 16 · p. 39
• SeamlessM4T matches NLLB-3.3B [NLLB Team et al., 2022] on T2TT X–eng and improves eng–X by 1 chrF++ point, while its low-resource Fleurs S2TT X–eng gain is +7.4 BLEU (40%) over AudioPaLM-2-8B-AST.The gain is +5 BLEU on low-resource directions supervised in AudioPaLM-2-8B-AST, suggesting improvement beyond supervision coverage from data quality and training recipes.Source: §4.4.2 · pp. 38–39, Table 17 · p. 39

4.4.3 Zero-shot Text-to-Speech Translation

• Zero-shot T2ST achieves 34.9 ASR-BLEU on 87 Fleurs X–eng directions versus 24.6 for S2ST, while eng–X T2ST scores 22.5 versus 21.5 for S2ST.The eng–X average excludes Bengali, Telugu, and Northern Uzbek because Whisper-Large-v2 ASR WER exceeds 100; the passage identifies non-English speech input as the more challenging direction.Source: §4.4.3 · p. 40, Table 20 · p. 40

4.4.4 Evaluation with spBLEU and Blaser 2.0.

• For S2TT, the authors report spBLEU and Blaser 2.0; Blaser 2.0 also scores S2ST across 82 X–eng and 35 eng–X directions, with eng–X averages also reported excluding three languages with WER above 100%.Including those three directions changes ASR-BLEU by -1.7 points, and the authors note Blaser 2.0 scores are less affected by ASR-model noisiness.Source: Table 18 · p. 40, Table 19 · p. 40, §4.4.4 · p. 41

4.4.5 Evaluation of X–X directions with spBLEU.

• The authors evaluate zero-shot Fleurs S2TT on X–X directions with spBLEU, averaging scores across 100 source languages for each target text language.The evaluation covers non-English-centric directions supported on both the source and target sides.Source: §4.4.5 · p. 41, Figure 9 · p. 41

4.5.1 Unsupervised speech pre-training

• Product quantization with 2 GVQ codebooks outperforms normal quantization with one; scaling training data and adding masked prediction objectives with 2 RPQ codebooks also improve performance.The ablation evaluates downstream X–eng S2TT on all 101 Fleurs directions, including 34 zero-shot directions.Source: §4.5.1 · p. 42, Table 21 · p. 42

4.5.2 Multimodal & multitasking X2T

• Adding T2TT to the multitasking loss improves T2TT by +1.8 BLEU on average across D1, D2, and D3; knowledge distillation improves S2TT by 0.6 BLEU points on average.Without T2TT loss, S2TT-only fine-tuning causes catastrophic forgetting of the pre-training T2TT task.Source: §4.5.2 · p. 43, Table 23 · p. 44
• 3× higher error rates result when ASR is evaluated zero-shot; mined S2TT data helps S2TT in M3, while adding speech-text-only data reduces T2TT accuracy.Joint fine-tuning with T2TT and knowledge distillation does not negatively affect ASR, where the teacher task is auto-encoding.Source: §4.5.2 · p. 43, Table 23 · p. 44

4.5.3 Leveraging Mined Speech-Text Data

• At most 400 hours of mined data in each direction yields the best average BLEU score; filtering SeamlessAlign by Sonar similarity can improve translation without increasing training compute.The experiments compare models adding 200, 400, or 600 hours in each direction, selecting top-ranked pairs by Sonar scores.Source: §4.5.3 · p. 44, Table 24 · p. 44

4.5.4 T2U pre-training in UnitY

• T2U pre-training helps UnitY converge faster, as reflected in ASR-BLEU scores after 10K updates, and makes fine-tuning more computationally efficient.The comparison initializes UnitY's T2U component with a pre-trained model versus leaving it without that initialization.Source: §4.5.4 · p. 44, Table 25 · p. 45

4.5.5 Leveraging mined speech-to-speech data

• +0.5 ASR-BLEU points for eng–X and −0.2 for X–eng result from adding mined S2ST data; speech generation quality improves slightly, so SeamlessAlign is included in final training.Source: §4.5.5 · p. 45, Table 25 · p. 45

Two-pass sequence generation

• Two-pass decoding retains end-to-end optimization while using cascade-style search, reranking, decoder biasing, or intermediate outputs; it makes optimization tractable and improves speech translation performance.Examples include searching for better outputs, reranking intermediate hypotheses, biasing them toward a domain, and providing streaming users an intermediate result.Source: §4.6 · p. 45

Codec-based audio modeling

• Codec-based units from EnCodec and SoundStream can be converted back to waveforms without an externally trained vocoder.The paper contrasts these units with acoustic units extracted from SSL-based audio representations such as XLS-R.Source: §4.6 · p. 45
• VaLLE [Wang et al., 2023a] used conditional autoregressive modeling of EnCodec audio for text-to-speech; VaLLE-X scaled language coverage with a cascade, while VIOLA [Wang et al., 2023c] explored cascade-free translation.Source: §4.6 · pp. 45–46

Multimodality & multitask for speech & text

• Source-side multimodality and multitask learning are distinct from two-pass learning [Anastasopoulos and Chiang, 2018]; multitask strategies share task information, with one-to-many speech translation and ASR and many-to-one T2TT and S2TT reported effective.Many-to-one training shares the decoder, while one-to-many training shares the encoder; task-dependent components have also been used to enhance individual task performance.Source: §4.6 · p. 46
• Joint representations across modalities aim to produce richer features and enable inter-modal tasks to benefit from shared training, with work spanning audio, vision, and audiovisual settings.Examples cited cover audio [Chen et al., 2022; Bapna et al., 2022; Zhang et al., 2023a; Rubenstein et al., 2023], vision [Chen et al., 2020; Gan et al., 2020; Fu et al., 2021], and audiovisual [Shi et al., 2022; Anwar et al., 2023] research.Source: §4.6 · p. 46

5.2 Human Evaluation

• Human assessment focuses on preserving speaker intention and the subjective quality of generated audio.The section describes the human evaluation process, including its protocols, data, and calibration.Source: §5 · p. 46, §5.2 · p. 48

5.1 Modality-Agnostic Automatic Metric: Blaser 2.0

• BLASER 2.0 uses SONAR embeddings to evaluate speech and text, supports 83 speech and 200 text languages, and offers a text-free option for evaluating speech outputs.Its supervised version predicts XSTS scores from source, translation, and reference embeddings; the unsupervised version averages source–translation and reference–translation cosine similarities. BLASER 2.0-QE is supervised and reference-free, using only source and translation embeddings when references are unavailable or questionable.Source: §5.1 · pp. 46–47
• The supervised BLASER 2.0 model outperformed the unsupervised model on each data partition; reference-free scores usually fell between them, except on new S2ST data.The unsupervised 2.0 model slightly outperformed its predecessor, while supervised v1.0 and v2.0 had the same average correlation with human judgments; v2.0 was used because it supports more languages. The authors hypothesize that references can diverge from sources due to segmentation or synthesis errors, or context-dependent non-literal translation.Source: Training · pp. 47–48, Table 26 · p. 47

5.2.1 Human Evaluation Protocols

• S2TT uses XSTS for translation quality, while S2ST uses XSTS for translation quality and MOS for naturalness.XSTS assesses semantic meaning preservation and was adapted for speech and text, including instructions to ignore non-speech tags, pauses, and non-speech noises. Three annotators scored each item; disagreements of 2 or more triggered two additional ratings, and the median was used. Moderated calibration used a cross-lingual calibration set to reduce language-level annotator biases.Source: §5.2.1 · p. 49

1. How clear is the speech?

• MOS annotators rated speech clarity, sound quality, and naturalness on a 1–5 Likert scale, using the same generations as XSTS except cross-lingual calibration items.Most languages had three annotators per generation, while Catalan, Finnish, Hindi, Indonesian, Italian, Korean, Portuguese, Romanian, Swahili, Telugu, Thai, Turkish, Urdu, and Vietnamese had one; MOS had no additional annotators or calibration when scores disagreed.Source: §1 · p. 50

5.2.2 Evaluation Framework

• Human evaluation used Fleurs test recordings and a shared set of sentences present in both languages, so S2TT and S2ST used identical items per language pair.Fleurs provides up to 350 sentences and three speaker recordings per language; the evaluated language sample balanced annotation availability, population coverage, and high- and mid-resource languages.Source: §5.2.2 · pp. 50–51

5.2.3 Human Evaluation Results

• For S2TT, SeamlessM4T-Large exceeded XSTS 315 in all 24 X–eng directions and 416 in all 24 eng–X directions, and beat Whisper-Large-v2 on aggregate X–eng scores.It significantly improved over Whisper-Large-v2 in 10 of 24 X–eng languages, with regressions in 10; only Japanese and Tagalog regressed by more than 0.5 XSTS.Source: §5.2.3 · pp. 52, 55, Table 29 · p. 52, Table 30 · p. 53
• SeamlessM4T-Large S2ST XSTS generally lagged Whisper-Large-v2 +YourTTS on X–eng, despite gains for Bengali, Telugu, and Swahili; most directions still scored at least 3.From S2TT to S2ST, its language-level XSTS scores decreased by 0.29 points on average, versus a decrease of 0.01 points for the baseline.Source: Figure 11 · p. 55, §5.2.3 · p. 56, Table 32 · p. 57
• SeamlessM4T-Large had more S2ST items at least 2 XSTS points behind its S2TT output, and its S2ST scores fell rapidly for source audio longer than about 8 seconds.In a 50-item inspection, half of the sampled SeamlessM4T-Large S2ST outputs had truncated S2TT endings.Source: §5.2.3 · pp. 58–59, Figure 12 · p. 58, Figure 13 · p. 59
• On X–eng speech output, SeamlessM4T-Large averaged 0.67 points higher in sound quality and 0.79 points higher in clarity than Whisper-Large-v2 +YourTTS.It surpassed the human reference in clarity and sound quality for all 8 measured X–eng languages; for naturalness, it performed better than the baseline in X–eng.Source: §5.2.3 · pp. 59, 61, Figure 14 · p. 60
• Blaser 2.0 correlated more strongly with XSTS than ASR-BLEU for S2ST and BLEU for S2TT, especially in eng–X directions, for both Pearson and Spearman coefficients.The metrics were compared at the language level using items evaluated with XSTS.Source: §5.2.3 · p. 61, Table 33 · p. 61, Figure 15 · p. 62

5.2.4 Limitations

• Human-evaluation limits included varying Fleurs sentence sets, few speakers and annotators, and MOS directions with single annotators and no calibration; median scores and bootstrap resampling addressed some uncertainty.XSTS calibration also sought to correct annotator bias, but recency bias and other human biases may persist.Source: §5.2.4 · p. 63

5.3.1 Robustness Against Background Noises

• On Fleurs with MUSAN noise and music, SeamlessM4T-Large outperformed Whisper-Large-v2 by averages of 33.3% for X–eng S2TT and 42.2% for ASR across noise types and levels.Source: Experimental framework · p. 64, Figure 16 · p. 64

5.3.2 Robustness Against Speaker Variations

• Speaker-variation robustness was evaluated without speaker subgroup labels using content-grouped chrFMS for normalized quality and CoefVarMS for quality variance.The metrics adapt Wang et al. [2020]’s label-free approach, using chrF for greater utterance-level stability than BLEU.Source: §5.3.2 · p. 65
• SeamlessM4T-Large outperformed Whisper-Large-v2 by an average of 49.4% on CoefVarMS and 18.3% on chrFMS across S2TT and ASR.The evaluation used label-free, content-grouped utterance-level chrF metrics on Fleurs languages with at least 40 content groups in the test sets.Source: §5.3.2 · p. 65, Table 34 · p. 65

6. Responsible AI / 6.1 Definitions / 6.3.1 Motivation

• Added toxicity means introducing toxic elements absent from the source, while gender bias is an incorrect gendered translation despite sufficient source clues; fairness concerns equitable content rather than voice robustness.For neutral inputs, a single gender choice is not enough to establish bias; the evaluation compares neutral English-to-X translations with translations of gender-inflected X-to-English inputs.Source: Toxicity · p. 66, §6 · p. 66, §6.3.1 · p. 73

6.2.1 Motivation / 6.2.2 Experimental Framework

• The proposed ASR-ETOX metric detects S2ST toxicity by applying ASR and ETOX with Toxicity-200 word lists, then counts an utterance as added toxicity only when output toxicity exceeds input toxicity.S2TT output is evaluated directly with ETOX; evaluation used Fleurs and HolisticBias, excluded four languages with segmentation or word-list consistency problems, and analyzed toxicity at sentence level.Source: §6.2.1 · p. 67, §6.2.2 · pp. 67–68

6.2.3 Results

• SeamlessM4T-Large reduced added toxicity by 63% versus Whisper-Large-v2 in Fleurs X–eng S2TT, 62% versus Whisper-Large-v2 + YourTTS in X–eng S2ST, and 26% versus Whisper-Large-v2 + NLLB-3.3B in eng–X S2TT.The authors suggest that filtering imbalanced toxicity in training data may have contributed to the improvement.Source: Figure 17 · p. 69, §6.2.3 · p. 69
• On HolisticBias, added-toxicity prevalence averaged 0.19% for S2TT and 0.13% for S2ST; SeamlessM4T-Large reduced S2TT added toxicity by 34% versus the cascaded baseline.S2TT affected 84 languages, while S2ST had added toxicity in 34 languages; less than 0.003% of S2TT outputs contained more than one added-toxicity token.Source: Figure 18 · p. 70, §6.2.3 · p. 70
• HolisticBias toxicity varied by demographic axis and language: body type was the largest dataset category, while some language distributions concentrated heavily in one axis, such as Bengali socio-economic status at 80%.Nonce and sexual-orientation categories had the highest reported concentration of toxicity for S2TT and S2ST, respectively, while age was lowest for S2TT and political ideologies lowest for S2ST.Source: Figure 18 · p. 70, Figure 19 · p. 71, §6.2.3 · p. 71
• Human review found 97–98% of detected HolisticBias items were true positives, whereas 81–84% of detected Fleurs items were false positives, reflecting the context and tokenization limits of word-list detection.The authors hypothesize that ASR before toxicity detection can miss toxic words, and note that blacklist token matching can fail on spelling variants.Source: §6.2.3 · pp. 70–72

6.2.4 Toxicity key findings and contributions

• SeamlessM4T-Large added toxicity ranged from 0.11% to 0.21% across modalities, directions, and datasets, with reductions of 51% on Fleurs and 34% on HolisticBias eng–X S2TT.Manual analysis confirmed that the toxicity mitigation also holds for real toxicity.Source: §6.2.4 · p. 72

6.3.2 Bias Experimental Framework / 6.3.3 Bias evaluation results

• The gender-bias study extended Multilingual HolisticBias to speech using synthesized inputs, covering 325 sentences across 19 languages, and examined gendered translations from neutral English inputs and performance gaps between gender-differing inputs.S2ST eng–X covered 17 languages and S2TT eng–X covered 25; chrF was used for S2TT, and measurements were repeated three times and averaged because TTS generation is nondeterministic.Source: §6.3.2 · pp. 73–74, §6.3.3 · p. 75

6.3.3 Bias evaluation results

• For eng–X, feminine references lowered SeamlessM4T-Large S2TT and S2ST quality in all languages except Thai; Catalan differences reached 10.3 chrF and 10.7 ASR-chrF points, respectively.The analysis interprets these differences as a preference for masculine target forms when the source sentence provides no gender information; masculine plurals can also be indistinguishable from generic plurals in some languages.Source: §6.3.3 · p. 76
• For X–eng, masculine-source translations scored better in 15 of 18 languages for S2TT and 14 of 18 for S2ST with SeamlessM4T-Large.The largest reported differences were 6.4 chrF points for Tamil in S2TT and 6.3 ASR-chrF points for Tamil in S2ST.Source: §6.3.3 · pp. 77–78
• On overgeneralization, SeamlessM4T-Large slightly outperformed Whisper-Large-v2 + NLLB-3.3B; on X–eng robustness, it trailed Whisper-Large-v2 in S2TT but outperformed Whisper-Large-v2 + YourTTS in S2ST.The reported masculine–feminine performance gap was generally more pronounced for overgeneralization than robustness, except in Blaser 2.0; ASR-chrF showed a higher percentage gap than Blaser 2.0.Source: §6.3.3 · p. 78

6.3.4 Gender data representation

• The English dataset analysis found masculine lexical representation predominant in most datasets, while EuroParl, Fleurs, and Flores had extremely low gender-marker representation.The authors describe the limited vocabulary as a scalability trade-off and suggest scalable dataset characterization could help balance datasets.Source: §6.3.4 · pp. 79–80, Figure 23 · p. 80

6.3.5 Bias Key Findings

• SeamlessM4T-Large favors masculine target forms by ∼10% on average and produces ∼3% lower-quality English translations from feminine inputs; analyzed datasets also overrepresent masculine lexica.The paper reports comparable bias results to the state of the art and says these findings support standardizing large-scale speech-translation bias evaluation.Source: §6.3.5 · pp. 80–81

6.4 Limitations

• The bias and toxicity analyses relied on string matching because suitable model-based techniques were unavailable for this multilingual multimodal setting, and the authors note substantial limitations.Toxicity lists can over-detect context-dependent terms and depend on tokenization; ASR adds false negatives, especially for eng–X. Gender noun lists use linguistic gender as a proxy and a limited 30-noun vocabulary, limiting generalizability.Source: §6.4 · p. 81

7. Social Impact & Conclusion

• A new LID model and SONAR-enabled speech mining produced more than 470,000 hours of automatically aligned speech translations for a unified model combining SeamlessM4T-NLLB, w2v-BERT 2.0, T2U, and HiFi-GAN.The system supports S2ST from 100 languages to English and from English to 35 languages, and S2TT in 100-to-English and English-to-95 directions. The authors open-sourced their contributions and released two model sizes for researchers with limited computing resources.Source: §7 · pp. 81–82
• 20% BLEU improvement over prior state of the art in S2TT, with XSTS scores above 4/5 for 24 evaluated English-source languages and gains over Whisper-Large-v2 in 7/24 into-English languages.The authors also report greater robustness than [Radford et al., 2022] to background noise and speaker variations.Source: §7 · p. 82

7.1 Augmenting world-readiness

• The authors argue that SeamlessM4T could ease multilingual speech exchange and broaden access for people with accessibility needs and languages with multiple writing systems.They envision it as a co-pilot for speech-heavy foreign-language interactions; speech translation may also help people who lack reading or writing skills or cannot rely on sight, and languages with script variance.Source: §7.1 · pp. 82–83

7.2 Future work

• The authors caution that ASR may vary across gender, race, accent, and language, while slang and proper-noun translation may be inconsistent across high- and low-resource languages.Source: §7.2 · p. 83
• Because live speech offers little opportunity to inspect or revise outputs, S2ST can raise interactional risks from mistranslations or toxicity; applications should augment, not replace, language learning or reliable interpreters, especially in legal or medical decisions.Source: §7.2 · p. 83
• Future work should preserve prosodic and emotional expressivity and develop low-latency streaming translation, which may support industry or educational adoption [Iranzo-Sánchez et al., 2022; Rybakov et al., 2022].Source: §7.2 · pp. 83–84

A. FAIRSEQ2

• FAIRSEQ2 is an open-source library of sequence-modeling components for text and audio tasks, distributed under an MIT license and available on GitHub.It provides building blocks for machine translation, language modeling, and other sequence-generation tasks.Source: A. FAIRSEQ2 · p. 106
• Its components include transformer implementations, streaming text and audio preprocessing for multi-terabyte training data, training utilities, and incremental beam-search generators.The streaming pipeline supports preprocessing, transformation, shuffling, and batching without explicit data-preparation steps or data-loading timeouts.Source: A. FAIRSEQ2 · p. 106
• Following FAIRSEQ, FAIRSEQ2 is designed for extensibility, with a library structure that supports component drop-ins, including components first written for FAIRSEQ.The authors expect the library to grow with components from the team and open-source community.Source: A. FAIRSEQ2 · p. 106
• Dependency inversion separates core and experimental components, allowing architectures to be modified with little code while plug-ins remain independent of parent blocks.Larger efforts such as UnitY and Sonar are maintained in separate repositories that use FAIRSEQ2 as a dependency.Source: A. FAIRSEQ2 · p. 106
• FAIRSEQ2 favors independent components over a self-contained pipeline, designed for use across varied training and inference environments and compatible with PyTorch and other frameworks.Users can build a complete pipeline, combine frameworks, or use a single component such as an optimizer.Source: A. FAIRSEQ2 · p. 106

B. Data Statistics

• Table 38 reports ASR and S2TT training-data hours for SeamlessM4T’s X2T models, distinguishing primary and mined data and English-to-language from language-to-English directions.Primary data includes open-source S2TT and pseudo-labeled ASR data.Source: B. Data Statistics · p. 107, Table 38 · p. 108
• Table 38 classifies languages as high-resource above 1,000 supervision hours, medium-resource from 500 to 1,000, and low-resource below 500; unsupervised languages are evaluated zero-shot.Source: Table 38 · p. 108
• Table 39 reports S2ST training-data hours separately for English-to-language and language-to-English directions.Source: B. Data Statistics · p. 107, Table 39 · p. 109

Model Card

• Meta AI Research developed SeamlessM4T-Large and SeamlessM4T-Medium on August 22, 2023, as multitasking UnitY models with Conformer speech and Transformer text and T2U components.Source: C. Model Card - SeamlessM4T · p. 110

Intended Use

• It supports ASR for 96 languages, speech-to-speech translation from 100 source languages into 35 targets, and speech-to-text translation from 100 source languages into 95 targets.The model card identifies researchers and the speech and text machine-translation research community as its primary users.Source: C. Model Card - SeamlessM4T · p. 110
• It supports text-to-speech translation from 95 source languages into 35 targets, text-to-text translation from 95 source languages into 95 targets, and speech synthesis for 36 languages.The model card also points to repository instructions and fine-tuning recipes.Source: C. Model Card - SeamlessM4T · p. 110
• The model is not released for production or intended for medical, legal, long-form, or certified translation; longer sequences may degrade in quality.It was trained on general-domain data and short text and speech inputs.Source: C. Model Card - SeamlessM4T · pp. 110–111

Metrics

• S2TT evaluation uses BLEU, spBLEU, and Blaser 2.0; S2ST uses ASR-BLEU and Blaser 2.0; T2TT uses chrF++; ASR uses normalized-text WER.SeamlessM4T-Large was also evaluated with human XSTS, added toxicity, robustness, and bias measures.Source: C. Model Card - SeamlessM4T · p. 111

Evaluation Data

• Evaluation datasets include Fleurs, Flores, CoVoST 2, CVSS, HolisticBias, and Multilingual HolisticBias; Fleurs provides n-way parallel speech and text in 102 languages.Source: C. Model Card - SeamlessM4T · p. 111

• The model was trained on parallel multilingual data from a variety of sources.Source: Training Data · p. 111

Ethical Considerations

• Translation access in low-resource communities could improve information access but also expose people with lower digital literacy to misinformation or online scams.The authors describe these scenarios as possible unintended uses if bad actors misappropriate the work.Source: Ethical Considerations · p. 111
• Training data mined from publicly available web sources may retain personally identifiable information despite extensive data cleaning.Source: Ethical Considerations · p. 111
• Toxic, biased, or false outputs could adversely affect people relying on translations for important decisions, particularly in health and safety.Source: Ethical Considerations · p. 111

• Researchers should consider additional integrity mitigations when using “added toxicity” in a research application.Source: Caveats and Recommendations · p. 111