PaperlayerView exact revision on arXiv

Source-linked AI summary

SeamlessM4T: Massively Multilingual & Multimodal Machine Translation

Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang

arXiv:2308.11596v3cs.CL

TL;DR

Unified speech translation remains less capable and less broadly covered than text translation, despite speech’s importance for multilingual communication. SeamlessM4T combines speech and text translation with ASR in one multilingual model, achieving strong benchmark results and evaluating robustness and safety. The authors release the models and supporting data and evaluation tools, while noting limitations in evaluation diversity and toxicity detection.

  • Problem

    Unified speech-to-speech translation has narrower coverage and lower performance than text translation, while existing systems often rely on cascaded pipelines and limited language directions.

  • Method

    SeamlessM4T combines ASR, T2TT, S2TT, T2ST, and S2ST using self-supervised speech representations, automatically aligned speech translations, and labeled and pseudo-labeled data.

  • Results

    SeamlessM4T improves translation quality across speech and text-related tasks, including 20% BLEU over the previous S2TT SOTA and 2.6 ASR-BLEU points over strong 3-stage cascades on S2ST.

  • Takeaways & Limitations

    The unified system broadens multilingual speech and text translation coverage while supporting robustness and safety evaluation and open-source use.

  • Takeaways & Limitations

    Evaluation is constrained by limited speaker diversity and small human-annotator samples, while word-list toxicity detectors can over-detect context-dependent terms and miss toxicity in speech outputs.

Abstract

from arXiv · show

What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed machine translation coverage beyond 200 languages, unified speech-to-speech translation models have yet to achieve similar strides. More specifically, conventional speech-to-speech translation systems rely on cascaded systems that perform translation progressively, putting high-performing unified systems out of reach. To address these gaps, we introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages. To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication

1. Introduction

SeamlessM4T addresses the limited coverage and performance of unified speech translation by combining multilingual speech and text capabilities in one system. It supports multiple translation and recognition tasks across broad language directions and reports strong benchmark results, responsible-AI evaluation, and open-source resources.

  • Research gap: Unified S2ST models lag behind text systems in coverage and performance, while existing systems emphasize high-resource languages, translation into English, and cascaded pipelines.Cascaded systems progressively combine ASR, T2TT, and TTS, whereas prior unified attempts did not match their performance.
  • Approach: SeamlessM4T supports ASR, T2TT, S2TT, T2ST, and S2ST within one unified model.The system performs S2ST from 100 languages to English and from English to 35 languages, with additional broad S2TT, ASR, T2ST, and T2TT coverage.
  • Results: 4.2 BLEU points is SeamlessM4T-Large’s improvement over AudioPaLM-2-8B-AST on Fleurs for S2TT into English.This corresponds to a reported 20% improvement over the previous end-to-end S2TT SOTA model.
  • Results: 2.6 ASR-BLEU points is SeamlessM4T-Large’s improvement over strong 3-stage cascaded models on Fleurs S2ST.On CVSS, it improves over a 2-stage Whisper-Large-v2 plus YourTTS cascade by 8.5 ASR-BLEU points.
  • Responsible AI: Added toxicity remains low at 0.11%–0.21%, while SeamlessM4T reduces it by 26%–63% versus state-of-the-art models.Gender-bias evaluation found masculine overgeneralization from neutral terms and limited robustness to gender variation, with comparable results to state-of-the-art models.
  • Resources: The project open-sources models, inference and fine-tuning tools, aligned-speech-data tooling, multilingual encoders, and the Blaser 2.0 S2ST evaluator.The released SeamlessM4T-Large and Medium models contain 2.3B and 1.2B parameters, respectively.

2. The Sociotechnical Dimensions of Multimodal Translation

Speech translation is prioritized because speech is central to communication, inclusion, accessibility, and script-independent interaction, yet research and systems remain predominantly text-centric. The section situates unified and cascaded approaches within this gap and identifies broader multilingual coverage, multimodal capability, and systematic evaluation as goals.

  • Why prioritize speech?: Speech can create stronger social bonds than text and remains a practical communication channel for people with limited literacy or visual impairments.The text also identifies script variance as a barrier for text-based communication and translation.
  • Cascaded systems: Cascaded speech translation chains ASR, T2TT, and TTS, allowing systems to reuse mature components but introducing cross-modal error propagation.The section notes that cascaded S2TT can underperform large-scale text-to-text translation, partly because transcription errors affect later stages.
  • Speech translation today: Prior speech models expanded language coverage rapidly but generally reported translation into English rather than translation from English to other languages.Whisper, AudioPaLM, and related systems are described as strong multilingual models with predominantly English-target evaluation.
  • Direct S2ST models: Direct S2ST research seeks to produce target speech representations without the full conventional cascade, but early systems lagged behind cascaded quality or decomposed the task into speech-to-unit and unit-to-speech stages.The section describes Translatotron and discrete-unit approaches as examples of this research direction.
  • Research goals: SeamlessM4T aims to unify S2ST, T2ST, S2TT, T2TT, and ASR while expanding both supported languages and translation directions.The stated goals also include systematic evaluation for safe, robust, equitable, and fair performance.

3. SeamlessAlign: Automatically Creating Aligned Data for Speech

SeamlessAlign is an open, large-scale multimodal corpus built by mining aligned speech–text and speech–speech segments across 37 languages. Its workflow combines improved language identification, over-segmentation, and Sonar-based joint embeddings, while data coverage and mining directions remain incomplete.

  • 3. SeamlessAlign: Automatically Creating Aligned Data for Speech: The mining pipeline projects speech and text into a joint embedding space to identify parallel speech–text and speech–speech pairs.Raw audio is chunked into overlapping segments, speech language identification is applied, and text comes from sentence-segmented NLLB data.
  • 3. SeamlessAlign: Automatically Creating Aligned Data for Speech: The pipeline adds an improved speech language identification model and a novel multimodal embedding space, expanding language coverage from 17 to 37 languages.The expanded system also increases the raw-audio amount to 4 million hours.
  • 3.2 Gathering raw audio and text data at scale: Over-segmentation uses voice activity detection, speech language identification, and overlapping candidate splits before mining selects the optimal boundaries.This strategy roughly octuples the number of potential segments considered.
  • 3.3.1 Sonar text embedding space: Sonar substantially outperforms Laser and LaBSE as a multilingual sentence representation, but its fixed-size bottleneck reduces translation BLEU by 7% into English and 13% out of English.The reported scores change from 35.2 to 32.7 into English and from 24.9 to 21.6 out of English.
  • 3.3.2 Speech encoders: The mining evaluation could not fully test speech encoders through end-to-end mining and translation loops, and some mining directions were left for future work.Mining was focused on 37 target languages rather than all 100 SeamlessM4T languages.
  • 3. SeamlessAlign: Automatically Creating Aligned Data for Speech: 470,000 hours of aligned speech data cover 37 languages in SeamlessAlign.The corpus includes more than 100 hours of English-speech alignments for nearly every language and exceeds 10,000 hours of English-text alignments for six languages.

4. SeamlessM4T Models

SeamlessM4T bridges speech and text translation with a unified architecture built from multilingual speech representations, text translation components, and jointly optimized UnitY modules. The models use staged training and large-scale or pseudo-labeled data to support multilingual speech and text tasks.

  • 4. SeamlessM4T Models: Direct speech-to-text translation models lag significantly behind strong cascaded models in large multilingual settings.This motivates building a stronger direct X2T model for translating both speech and text into text.
  • 4. SeamlessM4T Models: UnitY enables speech-to-speech translation through two passes: text generation followed by prediction of discrete acoustic units.Its components can be jointly optimized, unlike the independently chained components of cascaded systems.
  • 4. SeamlessM4T Models: The multitask UnitY model is fine-tuned in three stages, progressing from an English-target X2T model to a system supporting T2TT, S2TT, S2ST, and ASR.The framework integrates the SeamlessM4T-NLLB text model, w2v-BERT 2.0, and speech-output components.
  • 4.1 Unsupervised Speech Pre-training: w2v-BERT 2.0 combines contrastive learning with masked prediction over Gumbel vector quantization and random-projection quantizers.Its loss combines the contrastive, GVQ masked-prediction, and RPQ masked-prediction objectives.
  • 4.2 SeamlessM4T X2T: The X2T model uses separate Conformer-based speech and Transformer-based text encoders joined to a shared text decoder.It is trained on speech-audio and target-text pairs for speech-to-text translation.
  • 4.2 SeamlessM4T X2T: A re-trained 256K SentencePiece vocabulary raises coverage of the MTSU top 5K Chinese characters from 54% to 84%.SeamlessM4T-NLLB outperforms smaller NLLB-200 models and is on par with the larger 3.3B model across 95 languages.

4.3 Speech-to-Speech Translation

SeamlessM4T’s speech-to-speech system decomposes translation into speech-to-unit translation and unit-to-speech conversion, using discrete acoustic units, pretrained components, and multilingual data.

  • Architecture: The multitask UnitY model adds a T2U encoder-decoder and unit vocoder on top of X2T, with Stage3 training on S2ST data.
  • Architecture: S2ST is decomposed into speech-to-unit translation followed by unit-to-speech conversion using self-supervised discrete acoustic units.UnitY performs two-pass decoding: it first generates text, then predicts discrete acoustic units.
  • Acoustic units: K=10000 centroids from the 35th layer of XLS-R-1B achieved the best performance among tested acoustic-unit configurations.The tested centroid counts were 1000, 2000, 5000, and 10000.
  • Data: Text-to-unit models directly generate speech units for 36 target languages, avoiding dependence on language-specific TTS systems for pseudo-labeling.This approach addresses the difficulty of obtaining scalable, high-quality TTS models, especially for low-resource languages.
  • Training: The final UnitY fine-tuning used 121K hours of X–eng and eng–X S2ST data while freezing the X2T component and updating T2U.Freezing X2T was intended to preserve performance on tasks from earlier fine-tuning stages.

4.4 The SeamlessM4T Models

SeamlessM4T provides multilingual X2T and S2ST capabilities through Large and Medium models, with evaluations showing strong results against direct and cascaded baselines across tasks.

  • Models: SeamlessM4T-Large has 2.3B parameters and supports T2TT for 95 languages, ASR for 96, S2TT for 89 paired languages, and S2ST across English-centered directions.
  • Evaluation: SeamlessM4T-Large surpasses cascaded S2TT models by 2 BLEU points in X–eng and 0.5 BLEU points in eng–X directions.
  • Evaluation: SeamlessM4T-Large exceeds strong three-stage cascaded S2ST models by 2.6 ASR-BLEU points on Fleurs X–eng directions.
  • Evaluation: T2TT matches NLLB-3.3B in X–eng and improves eng–X by 1 chrF++ point, while low-resource S2TT improves by 7.4 BLEU over AudioPaLM-2-8B-AST.

4.5 Analysis and Ablations

Ablations show that representation quality, multitask objectives, mined data, and T2U pretraining each affect SeamlessM4T performance, with benefits depending on task and direction.

  • Speech encoders: Product quantization with 2 GVQ codebooks, scaled training data, and additional masked-prediction objectives each improved ablation S2TT performance.
  • Multitasking: Adding T2TT to the multitask loss improves T2TT by 1.8 BLEU on average across data setups and prevents catastrophic forgetting of the pretrained T2TT task.
  • Multitasking: Adding ASR data is crucial for ASR, because evaluating ASR zero-shot produces 3× higher error rates.
  • Mined data: Training with at most 400 hours of mined data per direction achieves the best average BLEU, indicating that filtering SeamlessAlign can improve translation quality.
  • UnitY: Pretraining T2U makes UnitY converge faster and is therefore more computationally efficient during fine-tuning.
  • UnitY: Adding mined S2ST data improves eng–X accuracy by 0.5 ASR-BLEU points but decreases X–eng accuracy by 0.2 points.

4.6 Related work

Related work positions SeamlessM4T within research on two-pass decoding, codec-based audio modeling, and multimodal or multitask learning for speech and text.

  • Two-pass decoding: Two-pass decoding preserves end-to-end optimization while inheriting benefits associated with cascading approaches.
  • Codec-based audio modeling: Codec-based audio modeling uses quantized audio autoencoders, including EnCodec and SoundStream, to represent audio for language-modeling systems.
  • Codec-based audio modeling: VaLLE introduced conditional autoregressive modeling of EnCodec audio data for text-to-speech synthesis in speech translation research.
  • Codec-based audio modeling: VaLLE-X expanded language coverage and translation through a model cascade, while VIOLA explored decoder-only codec-based translation without cascades.
  • Multimodality and multitask learning: Multimodal and multitask encoders aim to learn richer joint representations so inter-modal tasks can benefit from shared training.

5. Automatic and Human Evaluation

The evaluation combines modality-agnostic automatic metrics with human assessments of translation quality, naturalness, and robustness. BLASER 2.0 extends evaluation across speech and text, while XSTS and MOS assess semantic preservation and speech naturalness.

  • Modality-Agnostic Automatic Metric: BLASER 2.0: BLASER 2.0 evaluates speech and text outputs using SONAR embeddings, supporting 83 speech languages and 200 text languages.Its supervised variant predicts XSTS scores from source, translation, and reference embeddings; BLASER 2.0-QE removes the reference requirement.
  • Metric Training: BLASER 2.0’s supervised model uses annotated multimodal data, while its evaluation data include S2ST, S2TT, T2ST, and T2TT annotations.Audio longer than 30 seconds was filtered because SONAR encoders were not trained on long audio.
  • Human Evaluation: Human evaluation uses XSTS for semantic translation quality and MOS for S2ST naturalness, with three annotators examining each evaluated item.Moderated cross-lingual calibration is used to reduce language-level annotator biases and improve comparability.
  • Human Evaluation: XSTS is adapted across modalities to measure semantic meaning preservation for S2TT and S2ST outputs.The protocol requires modality-specific annotator instructions while retaining a modality-agnostic evaluation framework.
  • Human Evaluation: S2ST evaluation places greater cognitive demands on annotators than T2TT, making it slower to conduct.

1. How clear is the speech?

Human and robustness evaluations show strong S2TT quality and speech naturalness, but S2ST semantic scores lag behind a cascaded baseline and are affected by output audio problems. Performance also varies by direction, duration, and speaker conditions.

  • S2TT Human Evaluation: All 24 evaluated X–eng S2TT directions exceeded 315 XSTS, while all 24 eng–X directions exceeded 416 XSTS for SeamlessM4T-Large.Against Whisper-Large-v2 in X–eng, improvements were significant for 10 of 24 languages, with regressions above 0.5 limited to Japanese and Tagalog.
  • S2TT Human Evaluation: SeamlessM4T-Large outperforms Whisper-Large-v2 on average XSTS and on the percentage of X–eng sentences above XSTS thresholds 3 and 4.
  • S2ST Human Evaluation: SeamlessM4T-Large S2ST XSTS scores generally lag behind Whisper-Large-v2 +YourTTS, except for Bengali, Telugu, and Swahili.This contrasts with S2ST automatic metrics, where SeamlessM4T-Large consistently outperforms the baseline in X–eng.
  • S2ST Human Evaluation: S2ST XSTS decreases by 0.29 points from S2TT for SeamlessM4T-Large, versus 0.01 points for Whisper-Large-v2 +YourTTS.
  • S2ST Output Quality: 206 SeamlessM4T-Large S2ST generations had reported audio issues, compared with 22 Whisper-Large-v2 +YourTTS generations and 13 human references.Removing flagged items barely raised SeamlessM4T-Large XSTS, suggesting audio issues may correspond to the observed S2ST performance decrease.
  • S2ST Output Quality: Half of a 50-item inspection sample contained SeamlessM4T-Large S2ST outputs with truncated S2TT endings, which can substantially affect meaning.SeamlessM4T-Large also had a higher proportion of S2ST generations at least 2 XSTS points below their S2TT counterparts.
  • Duration Effects: S2TT performance generally declines for both models as source audio exceeds approximately 1 minute, although SeamlessM4T-Large is superior for most durations.
  • MOS Results: SeamlessM4T-Large has higher naturalness MOS than Whisper-Large-v2 +YourTTS in X–eng and averages 0.67 higher sound quality and 0.79 higher clarity of speech.For eng–X, naturalness exceeded 4 in Spanish and Swahili, while 11 directions exceeded 3.

6. Responsible AI

SeamlessM4T shows low added-toxicity prevalence across evaluated modalities, datasets, and directions, while reducing toxicity relative to cascaded and state-of-the-art baselines. Gender-bias evaluations reveal masculine preferences and lower translation quality for feminine references across most languages.

  • Toxicity evaluation: 0.11%–0.21% added toxicity varied by task and direction, with S2ST lower than S2TT for eng–X translation.For Fleurs, S2TT measured 0.21% and S2ST 0.16% in eng–X; X–eng measured 0.11% and 0.12%, respectively.
  • Toxicity evaluation: ASR before toxicity detection may create false negatives, helping explain lower detected toxicity in S2ST than S2TT.Manual inspection found similar toxic-word occurrences with minor differences between modalities.
  • Toxicity evaluation: 0.11%–0.21% added toxicity prevalence was observed across modalities, language directions, and datasets.Fleurs averaged 0.15%; HolisticBias averaged 0.19% for S2TT and 0.13% for S2ST.
  • Toxicity evaluation: 51% toxicity reduction was achieved across modalities and directions on Fleurs, compared with state-of-the-art systems.On HolisticBias, SeamlessM4T-Large reduced added toxicity by 34% for eng–X S2TT.
  • Gender-bias evaluation: Feminine references lowered translation quality in all languages except Thai for both S2TT and S2ST.Largest S2TT differences included Catalan at 10.3 chrF points, while S2ST differences included Catalan at 10.7 ASR-chrF points.
  • Gender-bias evaluation: When source gender information was unavailable, SeamlessM4T preferred masculine target forms, while analyzed datasets overrepresented masculine lexica.The reported average preference toward masculine forms was approximately 10% across both modalities.
  • Gender-bias evaluation: 15 of 18 languages showed better S2TT performance from masculine references for SeamlessM4T-Large.The largest masculine–feminine differences were 6.4 chrF points for Tamil and 5.0 for Urdu; Arabic instead favored feminine references.

7. Social Impact & Conclusion

SeamlessM4T addresses the limited scope of direct multilingual speech translation with a unified multimodal system and open research contributions. Its broader deployment implications include accessibility benefits, but also demographic variation, interactional risks, and unresolved expressivity challenges.

  • Conclusion: Direct speech translation still trails text translation in scope, while existing systems favor high-resource languages, into-English directions, and cascaded architectures.NLLB exceeds 200 text-translation languages, whereas unified speech-to-speech systems remain far less extensive.
  • Conclusion: SeamlessM4T combines S2ST, T2ST, S2TT, T2TT, and ASR using new multilingual data and modeling approaches.The system uses SONAR-based speech mining and integrates SeamlessM4T-NLLB, w2v-BERT 2.0, T2U, and HiFi-GAN.
  • Social impact: SeamlessM4T supports multilingual exchange and could provide more natural speech-based experiences than text-based counterparts.The authors connect these possibilities to on-demand multilingual communication and accessibility needs.
  • Limitations: SeamlessM4T’s benefits may vary across users because speech-recognition performance can differ by gender, race, accent, or language.The authors identify demographic variation as a distributional concern rather than claiming uniform benefit.
  • Limitations: Live speech translation offers fewer opportunities for checking or revising outputs, increasing interactional risks from mistranslations or toxicity.Back-translation or native-speaker review may be unavailable during live conversation.
  • Limitations: Natural S2ST remains an open challenge because speech includes prosodic and emotional components, while streaming translation requires further research.The paper identifies expressivity preservation and incremental translation as future research directions.

A. FAIRSEQ2

FAIRSEQ2 is an open-source, extensible library for sequence modeling over text and audio. It emphasizes scalable data processing, efficient training and inference, and separation between core and experimental components.

  • FAIRSEQ2: FAIRSEQ2 provides open-source sequence-modeling components for machine translation, language modeling, and sequence generation over text and audio.It is distributed under an MIT license and available on GitHub.
  • Features: Its components include transformer implementations, scalable text and audio preprocessing, training utilities, and optimized incremental beam-search generation.The data API supports streaming-style processing for multi-terabyte datasets.
  • Extensibility: FAIRSEQ2 is designed for extensibility, allowing components—including FAIRSEQ components—to be dropped into its library-like structure.The authors expect continued expansion by the project and open-source community.
  • Design principles: The library separates core and experimental code to avoid accumulating poorly supported or incompatible options in core functionality.This principle responds to the growth of mixed research code and conditional branches in FAIRSEQ.
  • Deployment environments: FAIRSEQ2 targets environments ranging from single-container training and edge-device inference to exaFLOPS-scale jobs and accelerated ASIC inference.The design acknowledges widely varying training and execution constraints.

B. Data Statistics

The supplementary data statistics describe the speech resources used to train SeamlessM4T’s ASR, S2TT, and S2ST components. They distinguish data sources, language directions, and supervision-resource levels.

  • Data statistics: Tables 38 and 39 report speech-audio training data in hours for X2T and S2ST models.Table 38 covers ASR and S2TT data, while Table 39 covers S2ST data.
  • ASR and S2TT data: The ASR and S2TT statistics distinguish primary open-source or pseudo-labeled data from mined data.The table describes primary data as open-source S2TT and pseudo-labeled ASR data.
  • Language directions: The statistics separate eng–X from X–eng directions for each language.This directional distinction appears in the ASR, S2TT, and S2ST data descriptions.
  • Resource categories: Languages are classified as high-resource above 1000 hours, medium-resource from 500 to 1000 hours, and low-resource below 500 hours of supervision.The resource labels are defined for the ASR and S2TT training-data statistics.

C. Model Card - SeamlessM4T

The model card presents SeamlessM4T as a research-oriented multilingual and multimodal translation model with broad task and language coverage. It documents evaluation practices and boundaries including general-domain, short-input use and non-production deployment.

  • Model overview: SeamlessM4T is a Meta AI Research multitasking UnitY model with Conformer and Transformer components for speech, text, and text-to-unit generation.The released variants are SeamlessM4T-Large and SeamlessM4T-Medium.
  • Documentation: The paper describing the exact training algorithm and data is identified as Seamless Communication et al., SeamlessM4T, Arxiv, 2023.The model card points readers to the paper for those details.
  • Capabilities: SeamlessM4T supports ASR for 96 languages, S2ST from 100 source speech languages into 35 targets, S2TT from 100 into 95, T2ST from 95 into 35, and T2TT across 95 languages.The listed capabilities cover multiple speech and text translation directions.
  • Intended use: The model card identifies researchers and the machine-translation research community as its primary users.Usage information and fine-tuning recipes are provided in the seamless_communication repository.
  • Scope boundaries: SeamlessM4T is a research model not released for production deployment and is not intended for medical, legal, domain-specific, or long-form translation.The model was trained on short text and speech inputs, so longer sequences may degrade in quality.
  • Evaluation: The model card lists BLEU, spBLEU, Blaser 2.0, ASR-BLEU, chrF++, WER, XSTS, added toxicity, robustness, and bias among its evaluation measures.FLEURS, Flores, CoVoST 2, CVSS, HolisticBias, and Multilingual HolisticBias are included among evaluation datasets.

Training Data

The model was trained using parallel multilingual data gathered from a variety of sources.

  • Parallel multilingual data from varied sources were used to train the model.

Ethical Considerations

The work prioritizes human users and seeks to minimize risks transferred to them. Expanding access through translation may also create vulnerabilities, particularly for low-resource communities and consequential decision-making.

  • Many studied languages are low-resource, so quality translation could improve information access for those communities.
  • Greater access may make people with lower digital literacy more vulnerable to misinformation or online scams.
  • Translation errors could adversely affect people who rely on translations for important health and safety decisions.

Caveats and Recommendations

The documentation recommends additional integrity mitigations for added toxicity in research applications and identifies the card template and licensing information used.

  • Researchers should consider additional integrity mitigations for added toxicity when using the model in research applications.
  • The model card uses the template from Mitchell et al. [2019] and cites a Creative Commons Attribution-NonCommercial 4.0 license.
Loading 2308.11596v3…