Is Prosody Lost in Translation? Fine-Grained Cross-Lingual Prosody Similarity Across Languages

Introduction

• Pitch and energy can convey emphasis, emotion, intent, and discourse structure beyond lexical meaning, so translations that lose these cues may misrepresent meaning or expression.Pitch shapes intonation, while energy contributes to emphasis, prominence, and emotional expression.Source: §1 · p. 1
• Existing S2ST systems often discard or simplify prosody, partly because prosodically parallel multilingual speech data are scarce, leaving prosody’s role and transferability underexplored (Ouali et al., 2025; Do et al., 2017; Jia et al., 2022, 2019, 2018; Inaguma et al., 2023; Kim et al., 2024; Lee et al., 2022).Recent expressive S2ST and prosody-transfer systems assume some cross-lingual correspondence, but its extent remains poorly understood (Swiatkowski et al., 2023).Source: §1 · p. 1
• Prior studies of inter-language prosody are few and mostly analyze utterance-level patterns, although meaning- and expression-bearing cues occur at finer temporal resolutions (Brannon et al., 2023; Avila and Ward, 2023; Zhou et al., 2024).The introduction motivates fine-grained analysis to understand which prosodic cues may be preserved across translation.Source: §1 · pp. 1–2
• The study uses professionally dubbed multilingual speech to characterize whether, where, and to what extent prosody is preserved across translation in realistic expressive settings.Dubbing aims to retain expressive and emotional characteristics while adapting speech to target-language constraints.Source: §1 · p. 2
• The paper presents a large-scale fine-grained analysis across English-German, English-Spanish, and English-French, comparing aligned word-level pitch and energy and examining temporal prosody patterns.This focus on local expressive patterns contrasts with prior utterance-level or monolingual analyses.Source: §1 · p. 2
• The analysis also examines how lexical word subgroups affect prosody transfer, with findings intended to inform multilingual prosody modeling, transfer, and controllable expressive speech generation.It characterizes aspects of prosody that are similar or dissimilar across languages.Source: §1 · p. 2

Expressive Speech-to-Speech Translation and Prosody Modeling

• Traditional speech translation commonly used cascaded ASR–MT–TTS pipelines [Jia et al., 2019], which prior work noted may discard paralinguistic information [Inaguma et al., 2023; Kim et al., 2024; Lee et al., 2022].Source: §2 · p. 2
• Direct S2ST approaches such as Translatotron 2 do not explicitly model cross-lingual prosodic dynamics, while evaluations emphasize translation quality, naturalness, intelligibility, and speaker similarity [Jia et al., 2022; Jia et al., 2018].Explicit fine-grained prosodic analysis remains limited.Source: §2 · p. 2
• SeamlessExpressive models pitch, duration, pauses, energy, and expressive conditioning embeddings, but often relies on indirect prosody modeling or coarse utterance-level supervision [Seamless Communication et al., 2023].Source: §2 · p. 2

Cross-Lingual Prosody Transfer and Multilingual Dubbing Analysis

• Studies report limited S2ST sensitivity to prosodic structure and limited understanding of cross-lingual prosody transfer [Swiatkowski et al., 2023; Zhou et al., 2024; Avila and Ward, 2023].Source: §2 · p. 2
• Human dubbing appears to preserve some expressive and prosodic characteristics across languages, but prior analyses mainly examine aggregate timing, speaking rate, lip synchronization, and broad prosodic statistics [Brannon et al., 2023].They do not provide fine-grained word- or contour-level cross-lingual correspondence.Source: §2 · p. 2

3.1 Audio Extraction and Quality Filtering

• The corpus uses professionally dubbed television speech across multiple genres, filtered for audio quality, single-speaker utterances on both sides, and semantic alignment to remove mismatched source–target pairs.Thresholds were calibrated on 100 randomly sampled German–English utterance pairs at each metric’s 75th percentile, retaining approximately the top 25% of examples by quality, then applied across all language pairs.Source: §3.1 · p. 3

3.2 Transcription and Word Alignment

• Word-level timestamps and bidirectional FastAlign links with intersection symmetrization support cross-lingual word matching despite syntactic reordering and translation variation.Transcripts are normalized before alignment, and OPUS TED2013 parallel data augments training to improve alignment robustness on short, noisy subtitle utterances.Source: §3.2 · p. 4
• FastAlign with OPUS augmentation achieves 82.2% alignment precision on the evaluated German–English benchmark.Alignment quality is measured using Alignment Error Rate on the RWTH German–English gold word-alignment benchmark derived from Europarl.Source: §3.2 · p. 4

3.3 Prosody Representation

• Word-aligned prosodic segments are reordered into corresponding sequences, resampled to 20 frames per word, concatenated, and mean-normalized for utterance-level analysis.Pitch is extracted with pYAAPT and missing values are linearly interpolated; energy is extracted with librosa. Constant-F0 traces are excluded, with no observed differences from retained segments in utterance length, word count, or alignment ratio.Source: §3.3 · pp. 4–5

3.4 Cross-Lingual Correlation Analysis

• The analysis correlates source and target prosodic sequences with Spearman rank correlation and uses shuffled-contour and leave-one-POS-group-out analyses to test structure and lexical contributions.The POS ablation removes frames for one English-tagged category from both sequences and recomputes correlation; a decrease indicates that category contributes positively to correspondence.Source: §3.4 · pp. 5–6

3.5 Temporal Feature Analysis

• A coarse temporal analysis compares source and target utterance duration and total phoneme count using Spearman rank correlation.Pairs with differing sentence counts are excluded to support utterance-by-utterance analysis; phoneme counts come from ASR transcripts, while durations come from word-level timestamps.Source: §3.5 · p. 6

4.1 Prosodic Correlation Analysis

• Pitch and energy correlate positively across all language pairs and exceed shuffled baselines; pitch correlations (0.229–0.246) are stronger than energy correlations (0.174–0.183).The aligned correlations significantly exceed the shuffled baseline (paired Wilcoxon, p < 10−80).Source: Table 4 · p. 6, §4.1 · p. 6
• Prosodic correlations are moderate but reflect broad positive correspondence; language-pair differences are small, and pitch correspondence is consistently stronger than energy correspondence.Pitch distributions have broader variance and stronger positive tails than energy, while language-pair effect sizes are negligible despite some statistically significant differences; EN–ES and EN–FR are slightly stronger for pitch than DE–EN, which is slightly stronger for energy.Source: §4.1 · pp. 6–7

4.2 POS Class Analysis

• Noun-associated regions most strongly affect measured prosodic correspondence, with noun removal reducing energy correlation by approximately 0.036–0.040 across language pairs.This pattern holds across both pitch and energy, and is especially strong for energy.Source: §4.2 · p. 7, Figure 4 · p. 8
• Function words increase energy correspondence when removed across all language pairs, while their pitch effects differ by pair and pitch effects vary more across utterances.For pitch, function words have positive effects in EN–ES and EN–FR but negative effects in DE–EN; verbs have weak or slightly negative effects, and modifiers remain near zero.Source: §4.2 · pp. 7–8

4.3 Temporal Feature Analysis

• Duration and phoneme-count correlations are consistently strong across language pairs, with duration correlations slightly stronger, indicating robust preservation of coarse timing structure.This correspondence is consistent with dubbing synchronization constraints and suggests actors need not drastically alter speaking rate.Source: §4.3 · p. 8, Table 6 · p. 8

5 Conclusion

• The work presents a fine-grained cross-lingual analysis using alignment-aware word-level contour analysis of pitch, energy, and temporal patterns in professionally dubbed multilingual speech.It examines German–English, English–Spanish, and English–French language pairs.Source: §5 · pp. 8–9
• Pitch and energy show positive cross-lingual correlations above shuffled baselines, with pitch correspondence generally stronger but more variable than energy.The authors suggest intonational structure may be preserved at a broader utterance level, while energy patterns depend more locally on lexical realization.Source: §5 · p. 9
• Noun-associated regions contribute most consistently to cross-lingual prosodic correspondence, while function-word patterns are less stable, particularly for energy.The findings indicate that correspondence varies across part-of-speech categories and may be stronger in content-bearing regions.Source: §5 · p. 9
• Coarse utterance-level temporal features correspond strongly across languages, with duration correlations slightly stronger than those for total phoneme count.These correlations are substantially larger than those for fine-grained pitch and energy contours, which are more sensitive to intonation, speaker style, and translation reordering.Source: §5 · p. 9
• The findings provide empirical insights for expressive speech translation and motivate research on prosody-aware multilingual speech modeling, controllable expressive translation, and cross-lingual speech generation.The authors emphasize alignment-aware analysis as important for future expressive speech translation systems.Source: §5 · p. 9

Limitations

• Professionally dubbed speech only approximates prosody transfer because actors may change timing, emphasis, or intonation for fluency and synchronization.These changes may limit how well dubbing reflects prosody transfer.Source: Limitations · p. 9
• The study examines pitch, energy, and coarse temporal features across three European language pairs, leaving pauses, rhythm, voice quality, and broader language generalization untested.Automatic transcription and bilingual word alignment may also introduce noise despite filtering; results may not generalize to typologically distant or tonal languages.Source: Limitations · p. 9
• Licensing restrictions prevent public release of the multilingual dubbing corpus, but the authors plan to release the full analysis pipeline for reproduction on compatible datasets.Source: Limitations · p. 9

A Additional Alignment Evaluation

• The benchmark compares five FastAlign symmetrization strategies using AER, precision, recall, total alignment links, and F-measure.It contains 508 manually annotated Europarl sentence pairs with sure and possible links.Source: A Additional Alignment Evaluation · p. 11, Table 7 · p. 11
• Precision matters more than coverage because incorrect links pair mismatched prosodic frames and corrupt the correlation signal, while missing links mainly reduce usable word clusters.Intersection is therefore preferred despite its moderately higher AER.Source: A Additional Alignment Evaluation · p. 11
• Intersection symmetrization achieves 82.2% precision and is used in the prosody pipeline, although grow-diag has the lowest overall AER.The evaluation uses the German–English gold alignment benchmark of 508 manually annotated Europarl sentence pairs.Source: A Additional Alignment Evaluation · p. 11, Table 7 · p. 11

B Additional Correlation Analysis

• Pairwise language-pair differences have negligible effect sizes (|Cliff’s δ| ≤0.031); DE–EN has slightly lower pitch but higher energy correlation.The small effects indicate limited practical differences and similar overall prosodic correspondence across the three analyzed pairs.Source: B Additional Correlation Analysis · p. 12
• Pitch correlations exceed energy correlations across all three language pairs (mean paired differences 0.046–0.070; 57–60% of utterances), with confidence intervals excluding zero and significant Wilcoxon tests.Episode-clustered bootstrap confidence intervals exclude zero for every language pair.Source: B Additional Correlation Analysis · p. 12

• Pitch ablation effects vary more across utterances, whereas energy effects are more consistent, particularly for nouns and function words.Source: C Additional POS Class Ablation Results · p. 12, Table 11 · p. 13
• Noun-associated regions have reliably positive ablation effects across all language pairs for both pitch and energy, while function-word energy effects are consistently negative.Source: C Additional POS Class Ablation Results · p. 12, Table 12 · p. 13
• Noun–function contrasts are robust for all energy analyses and DE–EN and EN–ES pitch, but less robust for EN–FR pitch under episode-level clustering.Source: C Additional POS Class Ablation Results · p. 12, Table 13 · p. 13

Temporal Resampling and F0 Interpolation Robustness

• 0.0008 was the maximum spread in mean per-utterance zero-lag Spearman correlations across 10–40 frames per word, supporting that using 20 frames per word does not affect the conclusions.Absolute correlations use a random subset of 3,000 utterance pairs per language pair; stability across settings is the relevant robustness measure.Source: D Temporal Resampling and F0… · p. 14
• For EN–ES, raw F0 contours yielded lower correlations than interpolated contours across frames-per-word settings; interpolation limits mismatch from language-specific voiced and unvoiced distributions.Without interpolation, measured correspondence can be disproportionately influenced by phonetic and voicing differences even when underlying intonational contours are similar.Source: D Temporal Resampling and F0… · p. 14

• Gender proportions were estimated per utterance with an automatic speech-based classifier, so the estimates are approximate (Doukhan et al., 2018).Source: E Gender Distribution · p. 14
• English and Spanish each had 57.7% male and 42.3% female speech.Source: E Gender Distribution · p. 14
• French had 54.8% male and 45.2% female speech, while German had 54.2% male and 45.8% female speech.Source: E Gender Distribution · p. 14