Seamless: Multilingual Expressive and Streaming Speech Translation

• Prior expressive S2ST work had not combined semantic translation with rhythm, pauses, and sentence-level voice-style preservation, while streaming work was largely S2TT, language-limited, or bilingual.The introduction identifies vocal identity, prosody, turn-taking, and timing as cues that contribute to natural communication and require balancing latency against translation quality.Source: §1 · p. 3
• SeamlessM4T v2 provides the multilingual foundation for the other models, supporting nearly 100 input languages and using UnitY2 and a w2v-BERT 2.0 encoder pretrained on 4.5M hours of audio.Its multitask model was finetuned with additional supervision from automatically aligned pairs to improve low-resource-language performance.Source: §1 · p. 3
• SeamlessExpressive translates from and into English in five languages while preserving vocal style and prosody, including underexplored features such as speech rate and pauses.It is built using commissioned and publicly available datasets.Source: §1 · p. 3
• SeamlessStreaming uses Efficient Monotonic Multihead Attention (EMMA) to produce low-latency translations before the source utterance is complete, supporting many-to-many ASR, S2TT, and S2ST at SeamlessM4T v2’s language scale.The introduction describes it as the first system to provide many-to-many simultaneous translation.Source: §1 · p. 3
• The evaluation combines automatic and human measures of prosody, rhythm, semantics, speech quality, latency, and translation quality, and tests robustness to noise and vocal-style variation.Measures include AutoPCP, a rhythm toolkit, XSTS, MOS, modified PCP, Ending Offset, Average Lagging, Length-Adaptive Average Lagging, BLEU, chrF, and Blaser 2.0.Source: §1 · pp. 3–4
• The authors combine SeamlessExpressive and SeamlessStreaming into the first publicly available system for real-time expressive cross-lingual communication and release its models and code.Their Responsible AI approach includes red-teaming, added-toxicity detection and mitigation, systematic gender-bias evaluation, and the inaudible localized SeamlessWM watermark.Source: §1 · p. 4

2.1.1 Meeting translation needs

• Participants relied on translation apps for everyday tasks but mostly used them for text, a disparity attributed to speech tools’ performance gap and users’ unfamiliarity with speech functions.For example, 26 of 34 participants translated subtitles rather than audio speech when watching news or television.Source: §2.1.1 · pp. 5–6

2.1.2 Real-time translation in synchronous contexts

• Interviews with 34 Mandarin- or Spanish-speaking participants with limited English proficiency found that consecutive speech translation was an inadequate workaround and that real-time interpretation could help in synchronous interactions.Participants described relying on family, friends, or coworkers when speech translation was unavailable, while linguistic dependency could hinder integration.Source: §2.1 · p. 5, §2.1.2 · p. 6

2.1.3 Expressive translation and the preservation of vocal style & prosody

• Participants most often defined natural S2ST as preserving vocal style and prosody, which communicate personality and intent; expressive real-time translation could support everyday social integration.Participants also warned that losing expressive cues, such as sarcasm, could misalign intent and cause misunderstandings.Source: §2.1.3 · pp. 6–7

2.2.1 Expressive systems

• Prior approaches had not unified semantic translation, rhythm, pauses, and voice style; this work builds aligned expressive data and reference-free prosody evaluation.Cascaded systems can use prosody labels passed through ASR, machine translation, and TTS, while voice-style transfer can use disentangled speech embeddings.Source: §2.2.1 · p. 7

2.2.2 Streaming systems

• Streaming research has emphasized S2TT and bilingual tasks; SeamlessStreaming combines SeamlessM4T v2 with EMMA for real-time many-to-many speech-to-speech and speech-to-text.The section identifies a need for streaming models that handle speech outputs and multilingual inputs and outputs, and that can build on improvements to their foundational models.Source: §2.2.2 · p. 8

2.2.3 The overarching goals of this effort

• The project targets end-to-end multilingual real-time speech translation with broader vocal-style preservation, expanded language coverage and directions, and systematic evaluation for quality, safety, equity, and fairness.Its goals include developing the data sets and foundational models needed for a unified system and supporting translations from English as well as into English.Source: §2.2.3 · pp. 8–9

2.3 Overview of Model Capabilities & Languages

• SeamlessM4T v2 covers 100 speech-input, 96 text-input, 96 text-output, and 36 speech-output languages; Expressive covers six languages, while Streaming matches v2 on ASR, S2TT, and S2ST.SeamlessExpressive supports English, French, German, Italian, Mandarin, and Spanish.Source: §2.3 · p. 9

3. SeamlessM4T v2

• SeamlessM4T v2 retains SeamlessM4T’s tasks and language set while its non-autoregressive text-to-unit decoder improves S2ST inference speed by 3x.The decoder decouples unit generation from output-length prediction, addressing hallucination or truncation associated with length mismatch and supporting real-time translation.Source: §3 · p. 12

3.1 Data for Speech Translation

• To address scarce paired speech data, SeamlessM4T v2 combines richer-task pretraining, automatic alignment, and ASR-data pseudo-labeling, following Seamless Communication et al. (2023).The richer tasks include T2TT with SeamlessM4T-NLLB and unlabeled-audio pretraining with w2v-BERT 2.0.Source: §3.1 · p. 13

3.1.1 SeamlessAlign

• SeamlessAlign expanded speech-encoder language coverage from 37 to 76 and added 114,800 aligned-data hours, with stronger S2TT performance on low-resource languages than Whisper.The encoders performed less effectively than Whisper for 23 mostly high-resource languages, but showed competitive fully zero-shot S2TT performance using fixed-size representations.Source: §3.1.1 · pp. 13–14

3.1.2 Pseudo-labeling

• For S2TT, available ASR data was pseudo-labeled with NLLB-3.3B, while S2ST training data was created by converting S2TT text into discrete units.The T2U model was trained on all 36 target speech languages; v2 used non-reduced (duplicated) units.Source: §3.1.2 · p. 15

3.1.3 Filtering

• After toxicity, length, special-character, and language-identification filtering, training data totaled 351K hours for S2TT and 145K hours for S2ST.Length filters excluded utterances shorter than 0.1 seconds or longer than 50 seconds, and text exceeding 250 subwords.Source: §3.1.3 · p. 15

3.2.1 Self-supervised speech representation

• The w2v-BERT 2.0 speech representation was pretrained on approximately 4.5M hours of unlabeled audio, and v2 removed the random-projection quantizer used in v1.The model has 24 Conformer layers and approximately 600M parameters.Source: §3.2.1 · p. 16

3.2.2 X2T: Into-text tasks

• The X2T model fuses a pretrained speech encoder with NLLB text components and uses knowledge distillation to jointly support speech translation, text translation, and ASR.Training proceeds in two stages: first supervised English ASR and into-English S2TT, then English-to-X S2TT and non-English ASR.Source: §3.2.2 · pp. 16–17

3.3 Predicting Units with UnitY2

• UnitY2 replaces UnitY’s autoregressive unit decoder with non-autoregressive generation, hierarchically upsampling subwords to characters and then units.It uses duration prediction supervised by a multilingual aligner and span-based glancing training; its non-reduced unit sequence preserves repeated units.Source: §3 · p. 12, §3.3 · pp. 17–18

3.3.1 Hierarchical subword-to-unit upsampling

• UnitY2 upsamples subword representations first to character length, then unit length, adding character and unit positional information and predicting character durations for unit generation.The duration predictor is trained with mean squared error against aligner-predicted durations in the logarithmic domain.Source: §3.3.1 · pp. 18–19

3.3.2 Unsupervised multilingual character-to-unit alignment learning

• To avoid unavailable external forced aligners for low-resource languages, UnitY2 uses an unsupervised character-to-unit aligner trained on 35 languages with curriculum learning; its released tool supports all 36 target languages.The alignment prior is disabled after 8k training steps, and monotonic alignment search extracts hard alignments.Source: §3.3.2 · p. 19, §3.5 · p. 24

3.3.4 Training UnitY2’s NAR T2U and aligner; 3.4 S2ST Training Setup.

• The NAR unit decoder and aligner are jointly trained with an intermediate character-level CTC loss to accelerate convergence.The S2ST setup uses separate teacher and student NAR T2U models, then finetunes only the student T2U weights on 145K hours of pseudo-labeled and aligned X–eng and eng–X data; the model converges in less than an epoch.Source: §3.3.4 · p. 20, §3.4 · p. 20

3.5 Results and Discussion

• In Fleurs, v2 improves S2TT X–eng by 10% over SeamlessM4T-Large and more than 17% over the strongest cascade; S2ST gains exceed 15% X–eng and reach 25% eng–X over v1.Against the strongest cascaded S2ST models, gains are 25% in X–eng and 15% in eng–X; CVSS shows a similar trend. In X–eng S2TT, v2 also outperforms direct models by more than 35%.Source: Table 6 · p. 21, §3.5 · p. 22
• On Fleurs ASR, v2 reduces relative WER by 21% versus v1 and 56% versus Whisper-Large-v2; on Flores T2TT, it loses 1.6 chrF2++ in both directions but remains on par with comparable NLLB models.On Fleurs-54 ASR, v2 improves over v1 by 19% relative WER and comes within 0.4 WER of MMS-L61-noLM-LSAH; on Fleurs-60 it improves over Whisper-Large-v3 by 4.4% WER. On CoVoST2 it gains 2.5 BLEU X–eng and 1.1 BLEU eng–X over v1, though it trails AudioPaLM-2-8B-AST by 1.2 BLEU in X–eng.Source: §3.5 · pp. 22–23, Table 7 · p. 22
• In zero-shot Fleurs T2ST, v2 gains 1.8 ASR-BLEU X–eng and 5.8 eng–X over v1, while low- and medium-resource languages improve across S2TT, S2ST, and ASR.Against cascades, zero-shot performance is on par with NLLB-3.3B + YourTTS in X–eng and exceeds NLLB-3.3B + MMS by more than 3.9 ASR-BLEU in eng–X. Low-resource gains are 2.8 BLEU, 4.3 ASR-BLEU, and -7.5 WER; medium-resource gains are 3.0 BLEU, 4.5 ASR-BLEU, and -4.7 WER, respectively.Source: Table 8 · p. 23, §3.5 · p. 23, Table 9 · p. 23

3.3.3 Efficient span-based glancing training for NAR unit generation

• Character-level input with non-reduced units works best for NAR T2U, whose best model beats the best AR model by 35% ASR-WER; GLAT, span-based masking, and character-level InterCTC improve intelligibility without an ASR-WER penalty from efficient single-pass GLAT.Efficient span-based GLAT masks character positions and maps them to aligned unit spans, estimating prediction accuracy from previous steps to avoid a duplicate forward pass.Source: §3.3.3 · p. 20, §3.5 · p. 24, Table 10 · p. 24

4.2 Expressive Modeling

• SeamlessExpressive combines Prosody UnitY2 for phrase-level rhythm, rate, and pauses with PRETSSEL for tone, emotion, and vocal style, retaining the backbone’s semantic translation quality.The system incorporates expressivity embeddings in both the translation model and speech generator.Source: §4 · p. 25, §4.2 · p. 31, Figure 7 · p. 32

4.1 Expressive Speech-to-Speech Translation Data

• Commissioned mExpresso expands six expressive read-speech styles into five languages, while mDRAL extends Ward et al. (2023) with multilingual spontaneous dialogues and more natural, less performed emotion.mExpresso recordings were translated and re-recorded by gender-matched native speakers, then reviewed for expressiveness and recording quality.Source: §4.1 · p. 25, §4.1.1 · p. 25, §4.1.2 · pp. 25–26

4.1.3 Automatically extracting expressive audio alignments

• SeamlessAlign was modified to rerank semantic neighbors using prosodic similarity; scaled alignment produced released SeamlessAlignExpressive metadata and pairs with less background-music mismatch, emotional imbalance, and singing.These properties were identified through manual inspection.Source: §4.1.3 · pp. 26–27
• On repeated-prosody mExpresso, p-xsim tuned the semantic/prosody tradeoff; adding prosody scoring improved alignment, and AutoPCP yielded higher-quality alignments than w2v-BERT embeddings.The benchmark tests recovery of expressive alignments, where repeated sentences with different prosody make semantic-only matching challenging.Source: §4.1.3 · pp. 26–27, Table 11 · p. 27

4.1.4 Extracting parallel segments from videos

• For multilingual videos, the pipeline split and merged speech segments before greedily matching bilingual candidates using text and speech similarity adjusted for time lag.It excluded segments longer than 25 seconds and pauses longer than 1.5 seconds, then filtered matches by similarity, duration mismatch, and time gap.Source: §4.1.4 · pp. 27–28

4.1.5 SONAR expressive

• Synthetic augmentation used SONAR Expressive to reconstruct speech units from semantic and speech-property encodings, and controllable TTS to synthesize pairs with chosen pause locations and speech rates.For cTTS, utterance-level speech-rate manipulation ranged from 70% to 130%; pause durations were randomly set between 0.3 and 1.5 seconds.Source: §4.1.5 · p. 28, §4.1.6 · pp. 28–29

4.1.7 Comparing across data sources

• Data sources had complementary tradeoffs: cTTS best aligned rate and pauses but lacked sentence-level emotion, while automatically aligned pairs mainly matched sentence-level styles and video pairs could differ in speaking rate.Commissioned datasets had the lowest speaker diversity; video alignment was theoretically expected to offer the best expressivity alignment, but timing and language differences limited it.Source: §4.1.7 · p. 29, Table 14 · p. 30

4.1.8 Training data pre-processing

• Training-data preparation applied denoising, silence removal, transcription, and vocal-style conversion as needed; Unit Voicebox used discrete speech units and prompts to transfer vocal style across aligned pairs.Style conversion was used for all datasets except SONAR Expressive, whose style was already mostly preserved; cTTS also received emotion and style variation through Unit Voicebox.Source: §4.1.8 · pp. 30–31

4.2.1 PRETSSEL: Expressive unit-to-speech generator

• PRETSSEL uses unsupervised speech-reconstruction pretraining to disentangle source-speech expressivity from linguistic units and generate target-language Mel features conditioned on source expressivity.Its textless acoustic model uses UnitY2-predicted durations, scaled from 20-ms units to 10-ms Mel frames, and its training losses require no text transcription or human annotations.Source: §4.2.1 · pp. 32, 34

4.2.2 Prosody UnitY2: Expressive speech-to-unit translation model

• Prosody UnitY2 injects source-speech expressivity embeddings into target-unit generation, addressing UnitY2’s difficulty preserving pauses and speech rate without acoustic conditioning.It adds the embedding to the subword-length T2U encoder output and FiLM-conditions the duration predictor and non-autoregressive unit decoder.Source: §4.2.2 · pp. 35–36

4.2.3 Expressive S2ST with SeamlessExpressive

• SeamlessExpressive cascades Prosody UnitY2 and PRETSSEL, conditioning target-speech synthesis on source expressivity and target-language embeddings before HiFi-GAN waveform generation.Source: §4.2.3 · p. 36

4.3 Experimental Setup

• The evaluation covers 10 English↔five-language directions and tests expressive translation on mExpresso, mDRAL, and FLEURS using content, vocal-style, prosody, speech-rate, and pause metrics.mExpresso and mDRAL assess expressivity alignment, while FLEURS assesses content preservation; the reported averages cover five X–eng and five eng–X directions.Source: §4.4 · pp. 38, 40

4.4 Results and Discussion

• Against Model 3, Model 5 gains +4.73 BLEU (mExpresso) and +1.59 BLEU (mDRAL) X–eng, and +4.14 BLEU (mExpresso) and +9.07 BLEU (mDRAL) eng–X, but trails by 1.34 BLEU on FLEURS X–eng.The models have comparable ASR-BLEU on FLEURS eng–X data.Source: Table 21 · p. 39, Table 22 · p. 39, Table 23 · p. 39
• PRETSSEL yields +0.22 vocal style similarity and +0.56 AutoPCP in X–eng, and +0.27 vocal style similarity and +0.4 AutoPCP in eng–X, with similar ASR-BLEU to Model 2.The comparison indicates that preserved expressivity did not severely degrade output intelligibility.Source: §4.4 · p. 40
• SeamlessExpressive (Model 5) improves content translation and preserves rhythm, tone, and vocal style over baseline SeamlessM4T v2 (Model 2), with consistent prosody gains across language directions.Source: §4.4 · pp. 40–41
• PRETSSEL has 65M parameters and RTF 0.014 versus Unit Voicebox’s 329M parameters and RTF 0.089; Voicebox has higher vocal style similarity, while AutoPCP is similar.The authors consider PRETSSEL a better fit when equipping SeamlessExpressive with streaming capability, given its smaller size and faster synthesis.Source: §4.4 · p. 41
• The expressive S2TT-plus-TTS cascade is sensitive to source-speech noise and has much lower ASR-BLEU than SeamlessExpressive; TTS better preserves vocal style, while SeamlessExpressive better preserves prosody.SeamlessExpressive has consistent gains in AutoPCP, speech rate, and pause metrics across language directions.Source: §4.4 · p. 41

4.5.1 Data ablation

• Parallel training data supports content translation quality, while prosody-aligned parallel datasets maintain prosody preservation; data-source ablations primarily affect ASR-BLEU.Source: Table 24 · p. 41, §4.5.1 · p. 41

4.5.2 Semantic and prosodic data filtering

• Higher semantic-quality splits raise ASR-BLEU by up to 8% versus low-quality data, while prosodic filtering improves rate and pause metrics without hurting ASR-BLEU or changing vocal-style similarity.These patterns held across language directions. The filtering study used multilingual video data, whereas the baseline was trained on more expressive data, so filtered results need not surpass it.Source: Table 25 · p. 42, §4.5.2 · pp. 42–43

4.5.3 Training ablation

• Joint training degrades vocal style similarity and AutoPCP, which the authors attribute to pseudo-parallel data’s weaker speaker and prosody alignment than human speech.The joint setup directs the NAR T2U decoder’s hidden states to PRETSSEL and uses source or target speech for PRETSSEL conditioning during training.Source: §4.5.3 · p. 43
• For X–eng, Bilingual, M2E, and M2M models perform similarly except in ASR-BLEU; for eng–X, M2M outperforms Bilingual and E2M in ASR-BLEU.Source: Table 27 · p. 44, §4.5.3 · p. 44

5.1 Efficient Monotonic Multihead Attention (EMMA)

• The streaming encoder feeds a simultaneous text decoder whose EMMA attention policy decides whether to emit a token or wait for more context.EMMA estimates monotonic alignments without supervision during training, with each attention head operating an individual policy (Ma et al., 2023; Raffel et al., 2017; Chiu* and Raffel*, 2018; Arivazhagan et al., 2019; Ma et al., 2020c).Source: §5.1 · p. 45

5.1.1 Numerical stable estimation

• EMMA uses a closed-form monotonic-attention estimate designed to be numerically stable and unbiased, addressing instability and bias in prior estimation.Its stepwise probability networks use learnable energy projections, a negative-initialized bias, and a temperature factor (Raffel et al., 2017; Ma et al., 2023).Source: §5.1.1 · p. 46

5.1.2 Policy regularization

• To prevent infinite-lookback monotonic attention from learning an offline policy, the model regularizes expected delay and alignment variance.The authors propose variance-based regularization because adding Gaussian noise to the stepwise probability network was inefficient, especially for speech translation models (Arivazhagan et al., 2019).Source: §5.1.2 · pp. 46–47

5.2.1 Fine-tuning from SeamlessM4T v2

• SeamlessStreaming is fine-tuned from SeamlessM4T v2 in two stages, then streams partial text and speech as source audio arrives.The first stage trains speech-to-text with a frozen speech encoder; the second freezes that component and trains the text-to-unit model. Inference updates the encoder on each audio chunk and synthesizes speech once unit chunks reach a minimum length; text latency uses AL and LAAL, while speech latency uses ending offset.Source: §5.2.1 · p. 47, §5.2.2 · p. 48, §5.2.3 · p. 48

5.3.1 Quality-latency trade-Off

• SeamlessStreaming performs ASR at much lower latency with less than 10 WER degradation from SeamlessM4T v2.The reported ASR output results are in Table 29.Source: §5.3.1 · p. 50, Table 29 · p. 50
• Speech-output quality drops more than text-output quality versus SeamlessM4T v2, partly because generated speech is discontinuous.The speech-to-speech results are presented in Table 30, and the comparison refers to text output in Table 28.Source: §5.3.1 · p. 50, Table 30 · p. 50, Table 28 · p. 50

5.3.2 Data resources

• High-resource languages show smaller quality drops and lower latency than lower-resource languages, while zero-shot settings show large quality drops and very small average lagging.The authors interpret the zero-shot pattern as an over-generation issue; Tables 31 and 32 report speech-to-text and speech-to-speech results, respectively.Source: §5.3.2 · p. 51, Table 31 · p. 51, Table 32 · p. 51
• For both text and speech output, English-related Italic and Germanic language groups have better quality and lower average lagging than more distant groups.Sinitic, Japanesic, and Indo-Aryan groups show larger quality drops and increased average lagging; Figures 13 and 14 cover text and speech output, respectively.Source: §5.3.2 · p. 51, Figure 13 · p. 52, Figure 14 · p. 52

6. Seamless

• Seamless unifies SeamlessStreaming and SeamlessExpressive as a publicly available real-time expressive speech-to-speech system.It focuses on sentence-level tone, emotional expression, and voice style rather than phrase-level speech rate and pauses, while retaining SeamlessStreaming’s semantic accuracy and latency.Source: §6 · p. 52

6.1 Architecture

• SeamlessStreaming produces streaming text and discrete units, which PRETSSEL converts into sentence-level expressive speech features before HiFi-GAN synthesizes translated speech in real time.The SeamlessStreaming component uses EMMA and a non-autoregressive T2U model.Source: §6.1 · p. 53
• Seamless-6 uses SeamlessExpressive’s high-resource language coverage, while Seamless-36 extends coverage to SeamlessStreaming’s language set using language-extended PRETSSEL.The extended PRETSSEL version was trained with 36 languages.Source: §6.1 · p. 53

6.2 Results and Discussion

• The section reports only automatic evaluations of Seamless.The authors also note that preliminary user testing did not indicate that omitting phrase-level preservation of speech rate and pauses harmed participants’ overall experience.Source: §6 · p. 52, §6.2 · p. 53

6.2.1 PRETSSEL extension

• Extending PRETSSEL from six to 36 languages maintains similar overall English-output quality but lowers non-English ASR-BLEU and vocal style similarity.The results indicate feasibility of broader language support, though a larger model may be needed for more capacity.Source: §6.2.1 · p. 53, Table 33 · p. 54

6.2.2 Quality-latency

• Seamless has a small ASR-BLEU degradation from SeamlessStreaming in both translation directions, lower ending offset, and a tradeoff between Seamless-6’s six target languages and Seamless-36’s broader coverage.Seamless-36 matches SeamlessStreaming’s coverage, while Seamless-6 performs better for SeamlessExpressive’s six target-language directions.Source: §6.2.2 · p. 54, Table 34 · p. 54, Table 35 · p. 55

6.2.3 Expressivity preservation

• Seamless’s vocal style similarity and AutoPCP decline relative to offline PRETSSEL because PRETSSEL uses partial context, with expressivity measured across latency settings.The measurements are reported for both to-English and English-to-target directions.Source: §6.2.3 · p. 54, Table 36 · p. 55, Table 37 · p. 56

7.1 Automatic Expressivity Metrics

• AutoPCP predicts human PCP prosody-similarity scores from XLS-R audio embeddings and showed stronger item- and system-level correlation than cosine similarity, with performance comparable to human-level judgments.The evaluation included item- and system-level Spearman correlations and RMSE; human-level performance compared one annotator’s judgment with the median of the others.Source: §7.1 · pp. 56–57, Table 38 · p. 57
• The rhythm toolkit measures speech rate and pause placement and duration; its speech-rate and pause similarity metrics moderately correlated with human PCP prosody-preservation judgments.The toolkit complements AutoPCP with interpretable measures of rhythm as realized in speech rate and pauses.Source: §7.1 · pp. 56–57, Table 39 · p. 58

7.2 Robustness Automatic Evaluation

• In speech-to-text robustness tests, SeamlessM4T-Large v2 improved over Whisper-Large-v2 by averages of 42% against noise and 66% against vocal-style variation.Against noise, its average improvements over Whisper-Large-v2 were 53.2% for X–eng S2TT and 50.5% for ASR; against vocal-style variation, it improved CoefVarMS by 66.4% and chrFMS by 31.5%. It also outperformed SeamlessM4T-Large on the reported tests.Source: §7.2 · pp. 58–59, Figure 16 · p. 58, Table 40 · p. 59

7.3.1 Human evaluation protocols

• Human evaluation used MOS for speech naturalness, sound quality, and clarity, and PCP for rhythm, emotion, overall expressive intent, and semantics; benchmark protocols were matched to test-set domains.MOS used a 5-point scale and three annotators per item; PCP used a 4-point scale, five annotators per item, and ignored speaker-voice differences. The test sets drew from mExpresso, mDRAL, and Fleurs, with MOS-quality measures across domains and PCP suited to expressive-style variation.Source: §7.3.1 · pp. 59–60, §7.3.2 · p. 60

7.3.3 Analysis and results

• Human results were partial: PCP covered only some X–eng directions, excluding Mandarin and French, with no eng–X results, while MOS covered all directions.Means and standard errors were estimated using 500 item-level bootstrap resamples. The human-evaluation SeamlessM4T v2 baseline differed slightly in unit-duration inference from the automatic-evaluation baseline, but the authors expected no impact on conclusions.Source: §7.3 · p. 59, §7.3.3 · p. 61

Key findings

• Adding PRETSSEL to SeamlessM4T v2 improved PCP scores across expressive dimensions but reduced some MOS subscores.Across X–eng directions, rhythm, emotion, and OEI rose by δ=0.30, δ=0.60, and δ=0.39, respectively; clarity and sound quality fell by δ=-0.49 and δ=-0.79, while naturalness was at parity (δ=0.03).Source: Key findings · pp. 61–62
• Against Model 3, SeamlessExpressive raised PCP rhythm (δ=0.58), emotion (δ=0.41), and OEI (δ=0.44), exceeding Human Reference on mExpresso but not mDRAL.The respective scores were 3.60 (0.07) vs 3.03 (0.14), 3.60 (0.08) vs 3.18 (0.13), and 3.47 (0.09) vs 3.03 (0.11).Source: Key findings · p. 62
• Across language and datasets, Model 5 had lower “naturalness” ratings (δ=-0.38; 3.62 (0.11) vs 4.01 (0.09)) and parity in “sound quality” and “clarity of speech” subscores.Sound quality was δ=-0.03; 3.60 (0.08) vs 3.64 (0.08). Model 5 performed worst overall on naturalness and sound-quality ratings.Source: Key findings · p. 62
• Unit Voicebox improved SeamlessExpressive’s clarity of speech (δ=0.26; 4.41 (0.06) vs 4.14 (0.07)) and sound quality (δ=0.55; 4.15 (0.07) vs 3.60 (0.08)), but not expressivity preservation.This comparison ablates the speech generation module by comparing Model 5 with Model 4.Source: Key findings · p. 62

7.3.4 Understanding MOS-quality measures for expressive models

• Expressivity modules including PRETSSEL and Prosody UnitY2 improve rhythm, emotion, and OEI measures but can lower clarity and sound-quality ratings, consistent with sensitivity to source-audio noise.The authors attribute this partly to PRETSSEL conditioning on reference audio and learning noise patterns associated with target audio during training.Source: §7.3.4 · pp. 62, 65
• Expressivity-preserving models retain source SNR, HNR, shimmer, and jitter more than SeamlessM4T v2, with PRETSSEL models preserving HNR most strongly.SeamlessM4T v2 target outputs were essentially independent of source inputs for these acoustic characteristics.Source: §7.3.4 · p. 64, Figure 17 · p. 64
• All models reduce noise to some degree, but less so in Models 3 and 5; those models also amplify negative HNR components, associated with more hoarse or breathy speech.The analysis compares source and target SNR, HNR, shimmer, and jitter values.Source: §7.3.4 · p. 64, Figure 18 · p. 65
• HNR and SNR positively associate with clarity and sound quality, while shimmer and jitter positively associate with naturalness but slightly negatively with sound quality.HNR has a substantial negative association with naturalness; the authors suggest modest breathiness or roughness may sound more natural if not pathological.Source: §7.3.4 · p. 65, Figure 19 · p. 66

7.3.5 Correlation between automatic metrics and human evaluation

• Expressive metrics strongly associate with each other, unlike expressive and semantic measures; AutoPCP correlates with OEI at ρ = 0.796, and speech-rate and pausing correlate with rhythm at ρ = 0.771 and ρ = 0.811, respectively.These associations support the validity of automatic and human measures, while expressive characteristics naturally co-vary.Source: §7.3.5 · p. 66

8. Responsible AI

• The work presents a multimodal, multilingual red-teaming methodology, measures gender bias with Multilingual HolisticBias, and digitally labels outputs with a robust watermarking module.The red-team effort was designed to quantify successful challenges for SeamlessM4T v2 and SeamlessExpressive.Source: §8 · pp. 66–67, Added Toxicity Mitigation · p. 70

8.1.1 Methods and implementation

• Red teaming tested text and speech outputs for critical errors, using five one-hour sessions with 24 employees and a dedicated interface for further challenges.The evaluation considered speech effects as well as lexical meaning; critical-error categories included safety concerns, opposite sentiment, and toxicity deviations.Source: §8.1.1 · pp. 67–68

8.1.2 Findings for SeamlessM4T v2

• Toxicity was the most prevalent red-team error for both models; added toxicity accounted for about 25% of SeamlessM4T v2 instances and 56% of SeamlessExpressive instances.For SeamlessM4T v2, 48% involved deleted toxicity; for SeamlessExpressive, 28% involved deleted toxicity. The prompts differed, preventing direct model comparison.Source: §8.1.2 · p. 69, §8.1.3 · p. 69, Table 49 · p. 70

8.2.1 Speech toxicity detection tools

• MuTox classifies toxicity directly from speech or text, avoiding dependence on ASR and potentially detecting prosodic, intonational, or emotional cues.The work also contributes a speech-toxicity dataset and benchmark spanning 21 languages.Source: §8.2.1 · p. 71, Methodology · pp. 71–72

Methodology.

• Supervised MuTox extends toxicity-detection coverage potentially tenfold while improving AUC by over 1% versus ASR-Detoxify.The reported comparison covers the eight languages available to Detoxify; the coverage increase is described as potential.Source: Methodology · pp. 72–73, Table 52 · p. 73
• Zero-shot MuTox loses only 4% AUC quality when tested on 19 languages, while supervised MuTox improves zero-shot performance by more than 7% on average.The reported results also state that zero-shot MuTox outperforms zero-shot ASR-MuTox on average.Source: Methodology · p. 72
• Almost 30% higher recall at fixed precision is reported for ASR-MuTox versus ASR-ETOX, while ASR-MuTox recall exceeds direct speech MuTox by >8%.The first comparison uses precision fixed at max(ASR-ETOX, 0.3), averaging 0.32.Source: Methodology · pp. 72–73, Table 53 · p. 74

8.2.2 Added toxicity mitigation

• Added-toxicity mitigation filters imbalanced training pairs and uses MinTox at inference to rerun translations with beam filtering when output toxicity is absent from the input (Costa-juss`a et al., 2023a).The reported evaluation covers S2TT and S2ST for SeamlessM4T v2 with and without MinTox, compared with SeamlessM4T-Large.Source: §8.2.2 · p. 73, §8.2.3 · p. 73

8.2.3 Added toxicity in SeamlessM4T v2

• MinTox consistently reduces added toxicity: ETOX reductions reach 80% versus the same model without MinTox and 90% versus SeamlessM4T-Large, while S2ST reductions reach over 50%.The lowest ETOX toxicity is consistently reported for SeamlessM4T v2 + MinTox; MuTox also favors it in most evaluated settings, but not Fleurs eng–X.Source: §8.2.3 · pp. 74–75, Table 54 · p. 77

8.2.4 Key Findings Summary

• MuTox benchmarks speech toxicity across 21 languages and supports zero-shot classification, expanding coverage potentially tenfold while improving AUC by 1% over lower-coverage predecessors.Compared with higher-coverage ETOX, MuTox improves recall by more than 30% at constant precision.Source: §8.2.4 · p. 76
• SeamlessM4T-Large v2 has low added-toxicity prevalence, with ETOX below 0.1% and MuTox below 3.5%, and reduces toxicity by up to 90% versus earlier SeamlessM4T-Large models.The key findings also report reductions versus SeamlessM4T-Large of up to 80% by ETOX and 35% by MuTox.Source: §8.2.4 · p. 77

8.3 Gender Bias

• SeamlessM4T v2 improves gender robustness over SeamlessM4T-Large by 0.4% in S2TT and 0.1% Blaser 2.0 in S2ST, but does not consistently improve overgeneralization and lags 2.2% in ASR-chrF.Source: §8.3 · p. 78

8.4 Localized Watermarking

• SeamlessWM enables frame-level identification of AI-generated audio, addressing prior watermarking approaches limited to one-second resolution and slow brute-force localization.The section also notes that many existing watermarking systems were designed to hide binary messages rather than trace provenance or detect generated content.Source: §8.4 · p. 80

8.4.1 Methodology

• SeamlessWM jointly trains a watermark generator and frame-wise detector, using dropped watermark windows to support localization and audio augmentations to improve robustness.The watermark is added to the audio; the detector produces a soft decision at each frame, and the training losses balance imperceptibility with localization.Source: §8.4.1 · pp. 80–81

8.4.2 Experiments & Results

• Perfect detection across all studied samples shows proactive watermarking outperforms passive detection; SeamlessWM also matches or improves WavMark detection and yields 37.82 dB SI-SNR versus 33.7 dB.The active-versus-passive comparison used a VoiceBox classifier and varied the proportion of masked phonemes; WavMark comparisons applied audio augmentations before detection.Source: §8.4.2 · p. 82, Table 57 · p. 82, Table 58 · p. 82
• 0.99 IoU versus WavMark’s 0.35 when one second of speech is AI-manipulated; SeamlessWM localizes altered segments at 1/16k-second frame resolution, versus WavMark’s one-second resolution.The comparison evaluates different proportions of watermarked speech mixed into non-watermarked speech.Source: Figure 29 · p. 83, §8.4.2 · p. 83

9. Social Impact & Conclusion

• Seamless combines SeamlessM4T v2, SeamlessExpressive, and SeamlessStreaming, with the latter two initialized from v2 and enabling expressive translation and low-latency streaming, alongside four safety measures.The measures are multimodal machine-translation red-teaming, added-toxicity detection and mitigation, gender-bias evaluation, and inaudible localized watermarking.Source: §9 · p. 83

9.1 Naturalistic Translations & Experiential Futures

• Seamless could support real-time multilingual dialogue in calls, AR/VR, streams, and wearables, with live captions available alongside speech outputs or on smart-glasses lenses.The proposed settings include multi-person interactions and locally run streaming interfaces.Source: §9.1 · pp. 83–84
• Seamless may ease everyday and professional communication for immigrants without being a panacea for social integration, and public release invites further technology-building for multilingual connection.The authors frame the system as a tool for softening language barriers and helping users pursue personal goals.Source: §9.1 · p. 84

9.2 Ethical Considerations & Future Work

• Model performance may vary with users’ race, accent, or gender, and ASR performance gaps can degrade expressive and streaming outputs, sometimes requiring users to alter their speech patterns.The authors note that user populations are heterogeneous despite fairness evaluations and checks.Source: §9.2 · p. 84
• Watermarking and a released detector may help identify synthetic content used for voice phishing or deepfakes, but safe deployment also requires AI literacy, scam prevention, public awareness, and industry standards.The authors describe watermarking as one part of a multifaceted response to malicious use.Source: §9.2 · p. 85
• Future work should expand language coverage, reduce high-/low-resource performance gaps, serve diverse users, and integrate sign languages and other visual signals for more adaptive and accessible translation.The paper notes that physical or circumstantial barriers can make some communication modalities unavailable.Source: §9.2 · p. 85

Contribution Statements; Acquisition

• Contributions were organized alphabetically by last name, with acquisition contributors handling annotations, vendor coordination, data licensing, and commissioning.The paper notes that the list cannot fully capture every contributor’s dedication and input.Source: Contribution Statements · p. 86, Acquisition · p. 86

Expressive

• Expressive work covered multimodal alignment, research and technical leadership, PRETSSEL and Prosody UnitY2 modeling, data strategy, evaluation, and controllable TTS generation.The listed work also included Auto-PCP and a rhythm toolkit.Source: Expressive · pp. 86, 88

Semantic

• Semantic work included data alignment, speech-encoder training, BLASER-v2, LID training, and human evaluation.Technical leads and contributors are identified for these efforts.Source: Semantic · p. 86

Fairseq2 and OSS; Demo Experiences

• Fairseq2 and OSS contributors built development, inference, fine-tuning, streaming-simulation, unity.cpp, and expressivity components, while demo teams delivered streaming and expressive experiences.The listed demo work spanned backends, model integration, user research, prototyping, and design.Source: Fairseq2 and OSS · p. 87, Demo Experiences · p. 87

Seamless (Expressive and Streaming); SeamlessM4T; Documentation

• Seamless and SeamlessM4T sections credit editors and leads for modeling, evaluation, RAI, narrative, data, and the unified streaming and expressive system.Documentation contributors prepared model and metric cards and coordinated research, streaming, product, and legal work.Source: Seamless (Expressive and Streaming) · p. 87, SeamlessM4T · p. 87, Documentation · p. 89

Foundational (M4T)

• Foundational M4T work covered pre-training and scaling, multilingual units and compression, X2T benchmarking, UnitY2 and NAR T2U, speech-to-speech ablations, NLLB retraining, and synthesis efficiency.The section also credits work on automatically aligned data ablations and multilingual speech synthesis.Source: Foundational (M4T) · p. 88

Streaming; Toxicity and bias; Safety; Linguistics and Social Impact; Leadership; Red-teaming

• Streaming contributions spanned TTS, encoders, multilingual modeling and inference; other teams led toxicity and bias research, watermarking, linguistics, social impact, project leadership, and red-teaming.The model card says Sonar was trained on parallel multilingual data from varied sources, with data selection and construction detailed in the paper.Source: Streaming · p. 88, Toxicity and bias · p. 89, Safety · p. 89

A. Model Card - Sonar Speech Encoders

• Sonar supports text encoders for 200 languages and speech encoders for 37, targeting research on parallel-data alignment, especially for low-resource languages.It was evaluated with BLEU on Fleurs, chosen for coverage of supported languages; it is not intended for domain-specific or document translation, and testing outside Wikimedia was limited.Source: A. Model Card - Sonar Speech Encoders · p. 101

B. Model Card - SeamlessM4T v2

• Developed by FAIR, Meta, and dated November 30, 2023, SeamlessM4T-Large v2 is a Multitask-UnitY2 model with Conformer and Transformer components.Its components include a Conformer speech encoder, Transformer text encoder-decoder, and Transformer encoder with a non-autoregressive decoder for T2U.Source: B. Model Card - SeamlessM4T v2 · p. 102
• The model is intended primarily for researchers, not production deployment, and is out of scope for domain-specific or long-form translation and certified translations.It was trained on short, general-domain inputs, and longer sequences may have degraded quality.Source: B. Model Card - SeamlessM4T v2 · p. 102
• It supports ASR for 96 languages, S2ST from 100 source speech languages into 35 target speech languages, and S2TT from 100 source speech languages into 95 target text languages.Source: B. Model Card - SeamlessM4T v2 · p. 102
• It also supports T2ST from 95 source text languages into 35 target speech languages, T2TT from 95 source text languages into 95 target text languages, and TTS for 36 languages.Source: B. Model Card - SeamlessM4T v2 · p. 102
• Evaluation datasets include Fleurs, Flores, CoVoST2, CVSS, HolisticBias, and Multilingual HolisticBias; Fleurs provides n-way parallel speech and text data in 102 languages.Source: B. Model Card - SeamlessM4T v2 · p. 102

Ethical Considerations

• Improved translation access in low-resource languages could also increase vulnerability to misinformation or online scams, while toxic, biased, or false outputs may affect important decisions, especially in health and safety.Source: Ethical Considerations · p. 102

Caveats and Recommendations

• Researchers using the model in research applications should consider additional integrity mitigations for added toxicity.Source: Caveats and Recommendations · p. 103

C. Model Card - SeamlessExpressive

• The model translates from five source languages into English and from English into five target languages, preserving rhythm, speech rate, pauses, and vocal style.Its stated primary intended use is expressive speech-to-speech translation.Source: C. Model Card - SeamlessExpressive · p. 104
• The model is intended for researchers, not production deployment or domain-specific medical and legal inputs, and its translations are not certified.Source: C. Model Card - SeamlessExpressive · p. 104
• Evaluation measures include ASR-BLEU for content preservation, AutoPCP, speech-rate correlation and pause alignment for prosody, and human PCP and MOS protocols.Source: C. Model Card - SeamlessExpressive · p. 104

Evaluation Data

• The evaluation data listed are mExpresso, mDRAL, and FLEURS; the work also used parallel multilingual speech from varied sources to train models.Source: Evaluation Data · p. 104
• The card warns that misuse could expose communities to misinformation or scams, and that toxic, biased, or false outputs could harm people relying on translations for important decisions, especially health and safety.Source: Evaluation Data · p. 104
• Researchers using the model should consider additional integrity mitigations for added toxicity.Source: Evaluation Data · p. 104

D. Model Card - SeamlessStreaming

• SeamlessStreaming supports streaming ASR in 96 languages, speech translation from 101 source languages, and text and speech outputs in 96 and 36 target languages, respectively.Its simultaneous translation algorithm uses Efficient Monotonic Multihead Attention.Source: D. Model Card - SeamlessStreaming · p. 105

Ethical Considerations

• The paper notes that low-resource communities may benefit from better translation access but could face greater exposure to misinformation and scams, while toxic, biased, or false outputs may affect important decisions.Potential harms are especially concerning for decisions involving health and safety.Source: Ethical Considerations · p. 105, E. Model Card - Seamless · p. 106

Caveats and Recommendations

• Researchers using Seamless in research applications should consider additional integrity mitigations for added toxicity.The model is a research model and is not released for production deployment.Source: Caveats and Recommendations · p. 105, E. Model Card - Seamless · p. 106

E. Model Card - Seamless

• Seamless supports simultaneous speech translation from 100 source languages into 6 or 36 target languages and preserves sentence-level prosody and vocal style.The card lists ASR-BLEU, vocal style similarity, AutoPCP, and ending offset as metrics, and Fleurs as evaluation data in 102 languages.Source: E. Model Card - Seamless · p. 106

F. Model Card - UnitY2 Aligner

• The UnitY2 Aligner extracts frame-level alignment between audio converted to discrete acoustic units and text converted to SPM tokens, outputting position-wise probabilities and a best-path alignment.Its intended users are researchers in speech translation.Source: F. Model Card - UnitY2 Aligner · p. 107

G. Model Card - AutoPCP

• AutoPCP is intended to compare prosodic properties of semantically equivalent utterances, including for evaluating expressivity-preserving translation and filtering parallel speech data.It is not intended for utterances with different semantic content, primarily non-speech audio, or comparisons across very different parallel audio corpora.Source: G. Model Card - AutoPCP · p. 108

Metrics

• AutoPCP performance is measured by RMSE against mPCP targets, item-level Spearman correlation, and system-level Spearman correlation.System-level correlation compares average model predictions with average targets.Source: Metrics · p. 108

Evaluation Data

• AutoPCP evaluation used labeled data resembling the training sources and distribution, and possible model biases were not specifically studied.Although it is trained to predict PCP scores from 1 to 4, predictions may exceed 5 for nearly identical audios; scores can also be affected by background noise and vocal-style similarity. Performance was evaluated only for six languages, though the model may generalize to some degree across the 53 XLSR languages.Source: Evaluation Data · p. 108

H. Metric Card

• Table 59 catalogs the automatic and human evaluation metrics used in the work.Its listed metrics are ordered mostly according to the paper’s narrative.Source: Table 59 · p. 109

I. SeamlessM4T v2

• Table 60 reports the parameter sizes of building components across SeamlessM4T v1 and v2 models.The table marks values that include parameters of the length adaptor.Source: §I · p. 110, Table 60 · p. 110

I.1 T2U Latency improvement in SeamlessM4T v2

• The latency evaluation compares v1 autoregressive T2U with v2 UnitY2 non-autoregressive T2U using 1,000 Fleurs S2ST audios and batch-size-1 decoding.Inputs were sampled from hin-eng, spa-eng, por-eng, rus-eng, and eng-eng; tests used one A100 and 96 CPUs, with beam width 5 for S2TT and autoregressive T2U.Source: §I.1 · p. 110
• SeamlessM4T v2 improves total S2ST inference speed by more than 3x over v1, with non-autoregressive T2U conversion time independent of input length.The comparison evaluates SeamlessM4T-Large with autoregressive T2U against v2 with UnitY2 and non-autoregressive T2U; v1 T2U decoding time scales linearly with generated text length and accounts for 70% of inference time.Source: §I.1 · p. 110, Figure 30 · p. 110

I.2 Data Statistics

• Tables 61–62 summarize per-language Sonar speech-encoder training and automatic-alignment data, with zero-shot Fleurs S2TT BLEU evaluation and Whisper-Large-v2 reference scores where supported.The statistics include raw audio for automatic alignment and human-provided ASR transcripts for encoder training.Source: §I.2 · p. 111, Table 61 · p. 112, Table 62 · p. 113
• Table 63 reports SeamlessM4T ASR and S2TT training hours by human-labeled, pseudo-labeled ASR, and automatically aligned data, distinguishing English-to-X from X-to-English.Languages with over 1,000 supervised hours are high-resource, 500–1,000 are medium-resource, and under 500 are low-resource; languages not supervised during the two finetuning stages are evaluated zero-shot.Source: §I.2 · p. 111, Table 63 · p. 114
• Table 64 gives SeamlessM4T S2ST training-data hours separately for English-to-X and X-to-English directions.Source: §I.2 · p. 111, Table 64 · p. 115

I.3 Detailed results

• Per-language scores across evaluated Fleurs tasks are reported, with result CSV files publicly shared.The detailed tables cover speech-to-text and speech-to-speech translation in both X–English and English–X directions.Source: §I.3 · p. 115
• Fleurs S2TT X–English results are presented in Tables 65–66 as BLEU scores.The tables are split into two parts.Source: §I.3 · p. 115, Table 65 · p. 116, Table 66 · p. 117
• Fleurs S2TT English–X results are presented in Tables 67–68 as BLEU scores.The tables are split into two parts.Source: §I.3 · p. 115, Table 67 · p. 118, Table 68 · p. 119
• Fleurs S2ST X–English results are presented in Tables 69–70 as ASR-BLEU scores.The tables are split into two parts.Source: §I.3 · p. 115, Table 69 · p. 120, Table 70 · p. 121
• Fleurs S2ST English–X results are presented in Table 71 as ASR-BLEU scores.Source: §I.3 · p. 115, Table 71 · p. 122

J.1 Data

• Evaluation-data statistics break down expressive S2ST data sizes by language direction for mExpresso, mDRAL, and FLEURS.FLEURS speaker-information entries are empty because speaker information is unavailable for that dataset.Source: §J.1 · p. 123, Table 72 · p. 123
• Unit Voicebox has 329M parameters, with two convolution layers and 24 Transformer layers using 1024-dimensional hidden states and a 4096-dimensional feedforward layer.Pretraining predicts masked spectrograms from speech and corresponding XLS-R units; data in each language was upsampled to match the amount of English speech.Source: §J.1 · p. 123
• Unit Voicebox was further finetuned on multilingual emotion data to augment expressive speech generation for Prosody UnitY2 training.Source: §J.1 · p. 123

J.2 Experimental Setup

• Full mExpresso dev and test results are provided in Table 73, alongside the mDRAL and FLEURS results in Tables 74 and 75.The supplied passages identify the tables but do not provide their result values.Source: §J.2 · p. 124, Table 73 · p. 124, Table 74 · p. 125

J.3 Semantic and prosodic data ablation

• The reported mDRAL and FLEURS results cover dev and test sets, while Figures 31 and 32 show test-set results for eng–X and X–eng pairs.The supplied passages give these datasets and language directions, but no numerical results.Source: Table 74 · p. 125, Table 75 · p. 126, Figure 31 · p. 127

K.1.1 Definition of operators

• Table 76 lists matrix operations and their PyTorch implementations.The passage provides the table's scope, not its operation entries.Source: Table 76 · p. 129

K.1.2 Detailed derivation

• The EMMA derivation rewrites the alignment equation to address denominator-related instability and computes alignment probabilities through matrix multiplication.It defines a transition matrix for reading between source positions without writing, then uses that matrix to compute α.Source: §K.1.2 · p. 129

K.2 Model Performance

• For eng–X FLEURS, Figures 33–36 cover SeamlessStreaming BLEU comparisons with SeamlessM4T v2 and latency for S2TT and S2ST.The supplied captions specify the evaluation scope but provide no plotted values.Source: Figure 33 · p. 131, Figure 34 · p. 131, Figure 35 · p. 131
• For X–eng FLEURS, Figures 37–40 cover SeamlessStreaming BLEU comparisons with SeamlessM4T v2 and latency for S2TT and S2ST.The supplied captions specify the evaluation scope but provide no plotted values.Source: Figure 37 · p. 132, Figure 38 · p. 132, Figure 39 · p. 132

• Table 77 reports the duration, in hours, of the PRETSSEL-36 extended pretraining dataset.Source: Table 77 · p. 133

M. Automatic and Human Evaluation

• The evaluation materials report source–target acoustic-noise correlations by model, correlations between PCP and automatic expressivity metrics, and aggregate PCP and MOS scores by language direction and dataset level.The tables also report standard errors for PCP and MOS scores.Source: Figure 41 · p. 133, Figure 42 · p. 134, Table 78 · p. 134

M.2 Prosodic Consistency Protocol (PCP)

• PCP compares paired utterances across semantics, emotion, rhythm, and overall expressive intent, with rhythm encompassing speed, pacing, and pauses.Overall intent combines rhythm, emotion, emphasis, intonation, and other features that shape an utterance’s impact; across languages, raters assess whether expressive qualities convey similar information.Source: §M.2.2 · p. 137, §M.2.3 · p. 138, §M.2.4 · p. 138

M.2.6 Task Description

• Raters listen to each segment from start to finish and score similarity, skipping unclear audio, focusing on shared content when lengths differ, ignoring leading or trailing silence, and setting voice differences aside.When content differs by more than a few words, raters score semantics as completely different and do not rate the other expressivity dimensions.Source: §M.2.6 · pp. 139–140

M.3.1 Task Instructions

• Assess similarity solely from vocal characteristics—overall resonance, pitch, power, and resulting speaker impression—rating shared characteristics from 1 (none; very different speakers) to 5 (nearly all; same person), disregarding words, meaning, emotions expressed, emphasis, intonation, and rhythm.The scale endpoints are “Not at all similar” (1) and “Extremely similar” (5).Source: §M.3.1 · p. 140, §M.3.3 · p. 141

N.1.1 Resourcing

• mExpresso reviewers check that prompted recordings are clear, free of background noise, and convey the assigned style or emotion; the project specifies nine styles, including fast, whisper, happy, and confused.The listed styles are Default, Enunciated, Fast, Whisper, Happy, Sad, Angry, Laughing, and Confused.Source: §N.1.1 · p. 142
• mExpresso recording guidance required faithful style and emphasis, the provided sentence, and clear audio, while rejecting incomplete, noisy, or otherwise unsuitable recordings.The rejection criteria also included non-native speech, mispronunciations, synthetic-sounding voices, and speech impediments or other voice-affecting conditions.Source: §N.1.1 · pp. 142–143

N.2.1 Resourcing

• Moderators led conversations and selected utterances for re-enactment, while qualification calls tested applicants’ language skills and prosodic re-enactment before approval or rejection.Applicants could decline, and approved participants received randomly generated file-identification tokens.Source: §N.2.1 · p. 143

N.2.2 Conversation

• Because participants struggled to re-enact prosody from native-language conversations in English, the live project used English conversations followed by target-language re-enactments.Participants were primarily native in the target language and strong in English; free-flowing discussions also moved beyond the initially chosen safe topics.Source: §N.2.2 · p. 144

N.2.3 Re-enactment

• Moderators selected suitable utterances from handwritten notes and repeated playback and re-enactment until the desired result, using moderators as substitute participants when needed.When a participant missed re-enactment, moderators aimed to reschedule; in the worst case, only the other participant’s re-enactment was retained.Source: §N.2.3 · p. 144

N.2.4 Quality Control

• 100% human QA checked original and re-enacted audio for emptiness and noise, verified transcripts, and supplied corrections before delivery.The checks were introduced after pilot fragments were found empty, noisy, or mismatched between original and re-enacted audio.Source: §N.2.4 · pp. 144–145
• Separate Zoom audio files obscured strong overlap during re-enactment until script-extracted fragments were reviewed; non-significant backchannels, laughter, and short responses were acceptable.Strong overlap was flagged, while fragments with overlap could be suggested for exclusion.Source: §N.2.4 · pp. 144–145