SEAM: Spectral Embedding Alignment and Merging for Low-Resource TTS

• Unseen phonetic or graphemic units require conditioning-space embeddings, while random initialization and extensive paired-audio fine-tuning are infeasible when target-language data is scarce or costly.Source: §1 · p. 1
• SEAM applies spectral filtering to subtoken embeddings and aligns the result to the pretrained embedding manifold with closed-form ridge regression, without target-language audio.Source: §1 · p. 1
• SEAM is designed to preserve temporal phonetic structure, suppress boundary artifacts, and improve data efficiency over standard initialization heuristics.Source: §1 · p. 1

• Prior cross-lingual and low-resource TTS approaches include speaker transfer, phoneme sharing, and meta-learning, but often assume substantial target-language data or use gradient optimization that can become unstable with extremely scarce data.Source: §2 · p. 1
• The authors describe vocabulary expansion as underexplored and present SEAM as an order-preserving, spectrally grounded initialization aligned with the pretrained manifold.Source: §2 · pp. 1–2
• Subtoken averaging is permutation-invariant and discards speech-relevant temporal order, leaving sequential structure to be relearned during fine-tuning and increasing sample complexity.Source: §2 · pp. 1–2

3 Method

• For a novel token decomposable into known subtokens, SEAM seeks a conditioning vector that is stable under generation, preserves temporal order, and aligns with the pretrained embedding manifold.Source: §3 · p. 2

3.1 Continuity Principle

• The continuity principle motivates smooth, order-preserving functions of constituent subtokens to limit distributional shifts during early fine-tuning.Source: §3.1 · p. 2

3.2 Spectral Merging Operator

• SEAM treats each embedding dimension as a discrete signal, applies a Gaussian low-pass filter to DFT coefficients, and retains initial frequency bins to suppress subtoken-boundary discontinuities.Source: §3.2 · p. 2
• Preserving complex DFT coefficients retains phase information encoding sequence progression, making the operator order-sensitive unlike mean pooling.Source: §3.2 · p. 2

3.3 Closed-Form Self-Calibration

• For calibration, SEAM pairs spectral features from valid old-vocabulary decompositions with pretrained embeddings, then solves a ridge-regression projection in closed form.Source: §3.3 · pp. 2–3
• For non-Latin scripts, the method transliterates a token into English IPA, segments it into known subtokens, projects the spectral features, and norm-normalizes the resulting embedding.Source: §3.3 · p. 3

4 Experiments

• Evaluation covers Russian, Kazakh, and French, uses Chatterbox as the primary backbone, and reports CER averaged across three random seeds on fixed 100-sample test sets.The study also includes native-speaker preference evaluation and tests a second architecture and dataset-split robustness in the appendix.Source: §4 · p. 3, §4.1 · p. 3

4.1 Experimental Setup

• All methods are fine-tuned under identical optimization schedules, and comparisons include random initialization with variance matched to pretrained embedding norm and the arithmetic mean of subtoken embeddings.Non-Latin scripts are transliterated using the transliterate library for English/Russian and qazaq_transliterator for Kazakh; French uses standard text normalization.Source: §4.1 · pp. 3–4

4.2 Sample Efficiency Under Fixed Budgets

• At 25h, SEAM improves over mean initialization by 2.22 CER points on Russian, 10.63 on Kazakh, and 0.87 on French, approaching or matching 100h baselines.At 100h, methods converge within ASR evaluation variance, making sample efficiency SEAM’s primary advantage.Source: §4.2 · p. 4, Table 1 · p. 4

4.3 Human Evaluation

• In blind side-by-side evaluation, five native speakers per language compared SEAM fine-tuned on 25h with mean initialization fine-tuned on 100h across three languages.Annotators judged overall quality as naturalness plus intelligibility and could select a tie.Source: §4.3 · p. 4, Figure 1 · p. 5

4.4 Sensitivity to Vocabulary Expansion Size

• Across expansions from 200 to 1,000 tokens, SEAM shows only marginal CER degradation, while random and average initialization deteriorate sharply beyond 500 tokens.Main experiments used 200 new tokens; SEAM’s relative advantage grows at higher expansion sizes.Source: §4.4 · p. 4, Figure 2 · p. 5

4.5 Extended Baseline Comparison

• On Russian with a 5h budget, SEAM outperforms order-aware baselines; Learned Attention reaches CER 15.30% but converges poorly with limited calibration data.Position-Aware Pooling and Attention Pooling report CERs of 23.37% and 20.14%, respectively; the passage attributes their weaker results to treating frequency components equally.Source: §4.5 · pp. 4–5, Table 2 · p. 5

4.6 Ablation: Order Sensitivity

• Reversal raises CER by 4.79 points (14.88% to 19.67%), while shuffling raises it by 8.03 points (14.88% to 22.91%).Shuffling also nearly doubles WER, from 34.41% to 53.66%; the authors attribute the larger effect to disruption of the precise temporal phonetic trajectory.Source: §4.6 · p. 5, Table 3 · p. 5

5 Conclusion

• SEAM spectrally merges subtoken embeddings and aligns them with closed-form regression, preserving phonetic order and reducing sample complexity by up to 4× across languages and architectures.The authors characterize the method as lightweight, reproducible, and intended to stabilize early-stage TTS adaptation.Source: §5 · p. 5
• SEAM’s use depends on subtoken decomposability and IPA transliteration.These are stated scope dependencies of the proposed vocabulary-expansion method.Source: §5 · p. 5

6 Limitations

• Reliable decomposition into known subtokens is required; mapping errors may propagate for non-Latin scripts or languages without robust IPA transliteration tables.Initialization does not remove the need for a suitable subtoken and IPA mapping.Source: §6 · p. 6
• SEAM does not replace downstream adaptation: acoustic alignment still requires paired target-language audio, though less than with other initialization methods.Source: §6 · p. 6
• At very small budgets, word-level overlap between training and test splits can inflate metrics non-monotonically, as observed for Kazakh.Source: §6 · p. 6

7 Ethical Considerations

• The authors release embedding-initialization code rather than full model weights and require explicit consent for deployments involving human voices.Experiments use publicly available, ethically sourced corpora; adopters are encouraged to follow local regulations and community guidelines.Source: §7 · p. 6

A.1 Continuity Principle and Acoustic Stability

• A Lipschitz-continuous conditional expectation motivates constructing new embeddings as smooth functions of their subtokens to limit acoustic deviations during early adaptation.The appendix states that preserving temporal order and spectral smoothness keeps embeddings close for phonetically similar tokens and stabilizes optimization from step zero.Source: §A.1 · p. 7

A.2 Order-Sensitivity Analysis

• SEAM's order-sensitivity guarantee is conditional: a retained non-DC frequency must be reversal-asymmetric, and the projection must not annihilate the resulting feature difference.When retained coefficients are reversal-invariant, such as only the DC component, the operator may map a sequence and its reversal to the same conditioning vector.Source: §A.2 · pp. 7–8
• The implementation sets K = 4 and σ = 2.0; for short sequences, K is an upper bound on the number of unique retained DFT coefficients.Source: §A.2 · p. 8
• For very short decompositions, Proposition 1's sufficient conditions may not hold; order sensitivity is then supported empirically by the order-perturbation ablation in Table 3.The low-pass argument is strongest when the effective retained-bin count is far below half the sequence length.Source: §A.2 · p. 8
• Cumulative spectral energy exceeds 99% by k = 3, supporting retention of bins 0–3 to preserve low-frequency phonetic structure while attenuating boundary artifacts.The Gaussian filter uses σ = 2.0; high-frequency components are suppressed by more than 85% by k = 4.Source: §A.2 · p. 8, Figure 3 · p. 8

B.1 Implementation details.

• Chatterbox is fine-tuned full-parameter for one epoch, while cross-architecture validation adapts VibeVoice with LoRA for three epochs and freezes its diffusion head.The configurations also specify distinct learning rates and effective batch sizes: 5 × 10^-4 and 64 for Chatterbox, versus 5 × 10^-5 and 16 for VibeVoice.Source: §B.1 · p. 8
• All runs use fixed random seeds and keep hyperparameters constant across languages; ridge regularization is λ = 10^-6 across languages and architectures.The paper describes this coefficient as minimal regularization for numerical stability, without language- or architecture-specific tuning.Source: §B.1 · pp. 8–9
• For non-Latin orthographies, the pipeline maps tokens to English IPA before decomposition; IPA strings are segmented with the original BPE tokenizer into pretrained-vocabulary subtokens.Russian and Kazakh use language-specific transliteration and phonological normalization; French normalization handles liaison, elision, and nasal vowels.Source: §B.1 · p. 9

B.2 ASR Evaluation Baseline

• French and Kazakh ASR baselines have CER <1%, while Russian’s 4.88% CER is treated as the floor for interpreting synthesized-speech error.The human-recorded-audio ASR results in Table 4 define the evaluation error floor.Source: §B.2 · p. 9, Table 4 · p. 9

B.3 Evaluation metrics.

• Evaluation reports CER, training-data counts with corresponding CER, and human Side By Side (SBS) judgments.The SBS measure is described in Section 4.3.Source: §B.3 · p. 9

B.5 Ablation: Sensitivity to Dataset Splits

• 14.97%–16.51% WER and 4.79%–5.67% CER across three Kazakh splits; mean WER was 15.61% and mean CER was 5.27%.The reported standard deviations were 0.66 for WER and 0.36 for CER, which the authors interpret as robustness to dataset composition.Source: §B.5 · p. 9
• From 5h to 25h, Kazakh CER rises for all initialization methods; the authors attribute the trend to lexical overlap differences between training and test utterances.At 100h, methods stabilize; SEAM’s lowest Kazakh CER is 4.39%, above the 0.98% ASR error floor.Source: §B.5 · p. 10

B.6 Ablation: Cross-Architecture Generalization

• 6.14% CER for SEAM on 5-hour Russian VibeVoice fine-tuning, versus 8.34% for average and 9.26% for random initialization.This cross-architecture result shows a performance advantage under low-resource fine-tuning.Source: §B.6 · p. 10, Table 5 · p. 10

B.7 Ablation: Audio-Guided Initialization

• The audio-guided variant first applies SEAM, then uses AdamW on the new embeddings and LM head with 1 hour of target-language calibration audio.The authors present this as an ablation of SEAM’s zero-audio design, not a comparison with a published audio-based initialization baseline.Source: §B.7 · p. 10
• Audio-guided refinement lowers initial Russian training loss to 7.2 from 10.2, but loss spikes to 15.0 and recovers only to approximately 10.0; zero-audio SEAM converges to approximately 4.0.The comparison is for Russian at a 5h budget, as identified in Table 6.Source: §B.7 · p. 10, Table 6 · p. 10
• The authors attribute the instability to inflated embedding norms that distort SEAM’s closed-form geometric alignment, and characterize zero-audio initialization as a deliberate design choice.They state that the disrupted calibration can leave the model in a poor local minimum requiring significantly more data to escape.Source: §B.7 · p. 10

C.1 Dimension 1: Realism (Naturalness)

• Annotators rate realism on a 5-point scale, from professional-speech-like naturalness to highly robotic or unintelligible output.The anchors distinguish naturalness and prosody from synthesis artifacts and intelligibility.Source: §C.1 · p. 11

C.2 Dimension 2: Linguistic Correctness

• Linguistic correctness covers target-language pronunciation, phoneme and stress accuracy, prosody, and fidelity to the reference text.Examples include language-specific phonological rules and penalties for word substitutions, omissions, or additions.Source: §C.2 · p. 11

C.3 Scoring and Preference Decision

• A model is preferred for significantly better realism or fewer linguistic errors; otherwise, raters may select a tie and are told to prioritize intelligibility.Annotators assess both samples against the criteria before making a final preference.Source: §C.3 · pp. 11–12
• SEAM preferences among non-tie SBS judgments are statistically significant for Russian (p = 0.041), Kazakh (p = 0.002), and French (p < 0.001).The analysis uses N = 125 judgments per language and 95% Wilson confidence intervals.Source: §C.3 · p. 12, Table 7 · p. 12

D Extended Related Work

• SEAM combines spectral low-pass filtering with closed-form ridge regression for zero-audio vocabulary expansion, addressing the order invariance of mean pooling and data requirements of contrastive alignment.The related-work discussion presents it as a complement to zero-shot and cross-lingual TTS frameworks.Source: D Extended Related Work · p. 12