PaperlayerView exact revision on arXiv

Source-linked AI summary

SwanVoice: Expressive Long-Form Zero-Shot Speech Synthesis for Both Monologue and Dialogue

Ruiqi Li, Yu Zhang, Changhao Pan, Ke Lei, Xiang Yin, Cheng Yang

arXiv:2605.30993v1eess.AS

TL;DR

Expressive long-form multi-speaker TTS remains difficult because turn-by-turn synthesis can disrupt conversational and acoustic continuity. SwanVoice combines speech-data processing with full-context zero-shot generation for 1–4 speakers, achieving higher richness and hierarchy scores than evaluated open-source baselines in both monologue and dialogue settings. Its main limitation is content accuracy.

  • Problem

    Expressive long-form dialogue TTS must preserve acoustic stability, speaker separability, affective continuity, and monologue quality beyond isolated-turn synthesis.

  • Method

    SwanVoice combines in-the-wild monologue and dialogue data processing with a 1–4-speaker zero-shot model using pause-aware text, speaker-turn conditioning, and staged training.

  • Results

    SwanVoice scores higher in richness and hierarchy than all evaluated open-source baselines in both monologue and dialogue settings, reaching 3.81/3.62 in monologue and 3.62/3.71 in dialogue.

  • Takeaways & Limitations

    The results support treating long-form dialogue as a full-context generation problem rather than a sequence of isolated turns.

  • Takeaways & Limitations

    Content accuracy remains weaker than the best baselines in several settings, and speaker switching can fail for acoustically similar speakers or short prompts.

Abstract

from arXiv · show

Zero-shot text-to-speech (TTS) has improved substantially for single-speaker synthesis, yet expressive long-form multi-speaker dialogue remains difficult. A common workaround is to synthesize each turn with a monologue TTS model and stitch the outputs together. This adds inference cost and often breaks acoustic consistency, conversational coherence, and affective continuity across turns. Recent dialogue TTS systems have begun to address this setting, but they still struggle to keep expressive coherence, controllable speaker switching, and monologue quality at the same time. We present SwanData-Speech and SwanVoice. SwanData-Speech builds monologue and dialogue corpora from in-the-wild audio, using Swan Forced Aligner for pause-aware word-level alignment and RobustMegaTTS3 for pronunciation-hard cases. Built on these data, SwanVoice is a zero-shot TTS model for 1--4 speakers, combining a 25 Hz VAE, raw-text conditioning with pause-aware symbols and pinyin substitution, and a flow-matching DiT with speaker-turn conditioning. Training starts from monologue speech, moves through mixed and real dialogue data, and then uses DiffusionNFT post-training with phone-level and speaker-similarity rewards. On SwanBench-Speech, SwanVoice obtains higher richness and hierarchy scores than all evaluated open-source baselines in both monologue and dialogue settings, while content accuracy remains the main limitation. Audio demos are available at https://swanaigc.github.io//#swanvoice.

1 Introduction

Long-form expressive dialogue TTS remains challenging because turn-by-turn synthesis disrupts conversational consistency, while end-to-end systems must preserve speaker identity, affective continuity, and monologue quality. SwanData-Speech and SwanVoice address these challenges through purpose-built data processing, pause-aware conditioning, speaker-turn control, and staged training.

  • Motivation: Single-speaker TTS has become reliable, but multi-party applications require treating conversations as one generation problem rather than concatenating independently synthesized turns.The workaround adds consistency challenges across turns.
  • Challenges: Long-form dialogue must stabilize acoustics, separate similar voices, preserve affective continuity, and avoid degrading monologue synthesis.These requirements expose failures less visible in short two-speaker generation.
  • Data bottlenecks: Dialogue data requires speaker-consistent segments, pause-aware transcripts, quality filtering, and sufficient non-neutral speech to learn affective variation.Speaker-split errors can corrupt turn control, while written punctuation can teach incorrect prosody.
  • SwanData-Speech: SwanData-Speech converts in-the-wild podcasts, radio dramas, and film/TV audio into monologue and dialogue subsets using alignment, vocal separation, and speaker segmentation.Its Swan Forced Aligner provides word-level timestamps and pause-aware annotation.
  • SwanVoice: SwanVoice supports 1–4 speakers with a 25 Hz VAE, raw-text conditioning, pause symbols, pinyin substitutions, speaker-turn IDs, and curriculum training from monologue to conversational data.The model uses a flow-matching DiT generator and subsequent post-training, as described in the passage.

2 Data Processing Pipeline: SwanData-Speech

SwanData-Speech converts a 2.59-million-hour Chinese–English raw collection into filtered monologue and dialogue datasets through enhancement, diarization, transcription, pause-aware alignment, and quality filtering. It supplements naturally collected data with pronunciation-hard synthetic examples while retaining raw text conditioning despite sparsity and pronunciation challenges.

  • Data collection: 2.59 million hours of raw audio—2.24 million Chinese and 0.35 million English—are processed into filtered monologue and dialogue subsets for SwanVoice.The collection combines mainly internal resources with selected open-source Chinese and English datasets.
  • Text representation: SwanVoice conditions on raw text to preserve semantic information, but this increases sparsity for rare and polyphonic characters and leaves pronunciation variants and code-switching as coverage challenges.Replacing all text with pinyin would reduce some sparsity but harm readability and authoring convenience.
  • Pronunciation-hard augmentation: RobustMegaTTS3 targets rare and ambiguous pronunciations using five generated sentences per dictionary entry, plus 20K Chinese, 20K English, and 100K code-switching hard cases rendered with MegaTTS 3.The hard cases cover polyphony, erhua, tone sandhi, stress shifts, irregular spellings, and mixed-language scenarios.
  • Hierarchical processing: The pipeline enhances audio, separates vocals, diarizes speakers, splits segments by speaker order, and independently applies ASR, punctuation refinement, and quality filtering to monologue and dialogue pools.Audio is preserved at its original sampling rate where possible and resampled to 24 kHz only at the final stage.
  • Pause-aware alignment: Forced alignment revises semantic punctuation according to acoustic pauses: gaps under 0.08 s are ignored, while longer gaps receive pause symbols or punctuation.This addresses weak correspondence between semantic punctuation and conversational pauses, especially at dialogue turn boundaries.

3 Method: SwanVoice

SwanVoice combines a 25 Hz VAE, raw-text and pronunciation-aware conditioning, speaker-turn representations, and flow-matching DiT generation for expressive monologue and multi-speaker synthesis. Its training uses a staged monologue-to-dialogue curriculum followed by reward-driven post-training for intelligibility and speaker similarity.

  • Acoustic Representation: The VAE compresses waveform inputs to 25 latent frames per second and uses HiFi-GAN decoding with multi-period, multi-scale, and multi-resolution adversarial discriminators.The encoder downsamples speech to reduce computation and facilitate speech–text alignment, while adversarial training targets high-frequency detail and perceptual fidelity.
  • Text and Speaker Conditioning: Raw BPE-tokenized text removes a separate G2P frontend, while pause tokens and 1,549 pinyin syllable combinations support natural pausing and controllable Chinese pronunciation.Random character-to-pinyin replacement improves robustness, and inference-time pinyin hints can enforce pronunciations for polyphonic characters.
  • Text and Speaker Conditioning: Speaker-turn labels align speaker identities with text tokens, and a lightweight Transformer processes padded text and turn embeddings before they interact with the speech latent in the DiT.The model conditions flow matching on processed text, speaker-turn embeddings, and a reference speech latent.
  • Training Curriculum: A three-stage curriculum progresses from approximately 2 million hours of Chinese and English monologue pretraining to mixed conversational training and real 2–4-speaker SFT.Monologue data remains in later mixtures to preserve monologue quality, while real conversations teach recording consistency and emotional coherence.
  • Post-Training: DiffusionNFT post-training optimizes generated samples with ASR-based robustness and speaker-similarity rewards through value-free online reinforcement learning.The rewards target intelligibility and recognition errors alongside timbre preservation, without differentiating through the recognizers.

4 Experiments

SwanVoice is evaluated on acoustics, semantics, and expressiveness using SwanBench-Speech protocols for monologue and dialogue. It leads open-source baselines on expressive richness and hierarchy, while content error remains a limitation.

  • Evaluation protocol: SwanBench-Speech evaluates acoustics, semantics, and expressiveness across timbre, reverb, fidelity, content error, prosodic coherence, richness, and hierarchy.Acoustic metrics include timbre consistency, reverb consistency, and SQUIM-PESQ sound fidelity; semantic metrics include CER/WER content error and SpeechJudge prosodic coherence.
  • Dialogue evaluation: Dialogue evaluation compares SwanVoice with six open-source long-form models, including ZipVoice-Dialog, MoonCast, MOSS-TTSD, FireRedTTS2, VibeVoice, and SoulX-Podcast.Expressiveness is judged through Emotional Variation, Vocal Dynamics, and Scene Appropriateness on a 1–5 scale using Gemini-3-Pro with randomized, system-blinded evaluation.
  • Monologue results: 3.81 richness and 3.62 hierarchy exceed all evaluated open-source baselines for monologue generation.Relative to VibeVoice, the strongest baseline on these metrics, the gains are 0.39 and 0.56 points, respectively.
  • Monologue results: 0.93 timbre consistency, 3.60 sound fidelity, and 3.56 prosodic coherence are at or above the open-source average, although SwanVoice is not best on content error.The monologue comparison includes ten open-source models, including VibeVoice, F5TTS, FishSpeech-1.5, and CosyVoice3-0.5B.
  • Dialogue results: 3.62/3.71 richness/hierarchy are 0.53/0.56 points above the strongest baselines in dialogue generation.Content error is below the baseline average but not the best in the table, and demonstrations include 3–4-speaker cases.

5 Conclusion

SwanVoice frames long-form dialogue as full-context generation and achieves higher richness and hierarchy scores than all evaluated open-source baselines in monologue and dialogue. Its main limitations are weaker content accuracy and unreliable speaker switching, motivating improvements in pronunciation, alignment, pause modeling, and speaker-turn conditioning.

  • SwanVoice treats long-form dialogue as full-context generation rather than isolated-turn synthesis.
  • SwanVoice obtains higher richness and hierarchy scores than all evaluated open-source baselines in both monologue and dialogue settings.
  • The data pipeline addresses failure modes through speaker-aware segmentation, pause-aware alignment, pronunciation-hard-case handling, and emotion-based filtering.
  • Content accuracy remains weaker than the best baselines in several settings, while speaker switching can fail with acoustically close speakers or short prompts.
  • Future improvements should target pronunciation control, alignment and pause modeling, and more robust speaker-turn conditioning to make long-form speech generation more reliable.

Appendices · A Swan Forced Aligner · A.1 Why Do We Need a Forced Aligner?

The appendix motivates Swan Forced Aligner by showing that readable ASR punctuation poorly reflects acoustic pauses, causing unreliable TTS pause control and supervision errors in expressive audio. It therefore calls for alignment grounded in speech evidence to recover boundaries and pause structure.

  • Appendices: SwanVoice targets expressive long-form zero-shot speech synthesis for both monologue and dialogue.
  • A.1 Why Do We Need a Forced Aligner?: ASR punctuation is optimized for readability and semantic plausibility rather than waveform pause structure.It may correlate weakly with real pauses, hesitations, and phrase boundaries.
  • A.1 Why Do We Need a Forced Aligner?: Unreliable punctuation-to-pause correspondence gives TTS models weak or inconsistent pause control.Punctuation may fail to trigger pauses, while pauses may occur without punctuation, degrading prosody and controllability.
  • A Swan Forced Aligner: A dedicated forced aligner is needed to ground textual units in speech and recover word boundaries and pause structure from acoustic evidence.
  • A.1 Why Do We Need a Forced Aligner?: Expressive in-the-wild recordings require reliable transcripts, temporal boundaries, and fine-grained attribute labels before serving as training supervision.Without them, annotation errors propagate into downstream generation models.
  • A.1 Why Do We Need a Forced Aligner?: Imperfect supervision complicates controllable audio generation because models must separate linguistic content, speaker identity, pronunciation, style, and expressive factors.

A.2 Overview

Swan Forced Aligner targets transcript-conditioned word-level alignment with accurate pause-aware boundaries for long-form speech. It combines explicit structural modeling and controllable decoding to improve interpretability, diagnosis, and efficiency over less integrated alignment approaches.

  • Motivation: Forced alignment must handle pauses, variable speaking rates, weak articulations, and annotation noise that can degrade word-level temporal boundaries.These challenges are especially relevant when downstream speech generation or annotation refinement depends on accurate pause-aware timing.
  • Contribution: Swan Forced Aligner combines an interleaved word/blank topology, structured decoding with calibrated unary and transition scores, and optional posterior-based decoding.The posterior-based mode addresses locally ambiguous evidence in noisy long-form speech segments.
  • Novelty: The model maintains an explicit monotonic alignment lattice, making decoding more controllable, interpretable, and diagnosable than direct timestamp prediction.It also models state emissions and transitions within one alignment framework rather than relying solely on frame classification.
  • Efficiency: Swan Forced Aligner is computationally efficient through compact parameterization, modest activation memory, and low-latency Viterbi decoding.This addresses the practical cost concerns associated with large autoregressive timestamp predictors and joint-sequence processing.
  • Motivation: Accurate time structure is a practical requirement because local timing errors can accumulate into content drift, unstable conditioning, and generated-audio mismatches.These long-form effects motivate evaluation beyond isolated frame-level fidelity, including consistency, preference, and synchronization over longer contexts.

B Method … B.5 State Scoring and Stability-Oriented Calibration

Swan Forced Aligner performs transcript-conditioned, word-level alignment by combining cross-modal acoustic and text representations with an explicit word–blank topology and calibrated monotonic decoding. Its stability-oriented scoring normalizes unary and transition evidence and independently calibrates their contributions.

  • B.1 Problem Setup: The task assumes a given transcript and localizes each word’s temporal boundaries despite pauses, speaking-rate variation, weak articulation, annotation noise, and uncertain labels.Available word annotations and confidence scores supervise frame-level occupancy and duration targets.
  • B.2 Lexical Word Representation: Each lexical word is tokenized into subwords, then followed by <|wbd|> so word-level alignment targets match tokenizer-level representations.The anchor aggregates the preceding subword span into a lexical-word state, avoiding independent alignment of every subword token.
  • B.3 Backbone Encoders: Pretrained acoustic and text encoders produce frame-level and word-anchor representations, with cross-modal conditioning providing alignment cues before structured decoding.Word anchors summarize complete lexical words, including words split into multiple tokenizer pieces.
  • B.4 Structured Alignment Topology: The latent alignment space interleaves initial, internal, and final blanks with word states, explicitly representing occupancy, pauses, silences, and transitional gaps.Internal blank prototypes depend on neighboring word states, while boundary blanks use separate learnable parameters.
  • B.4 Structured Alignment Topology: Monotonic paths use stay, one-state advance, and blank-skipping transitions, enforcing transcript-consistent decoding while modeling different alignment movements.The transitions support short coarticulatory gaps, long pauses, and phrase-level boundaries.
  • B.5 State Scoring and Stability-Oriented Calibration: Frame–state unary scores combine acoustic and state representations, while destination-conditioned transition heads and pairwise skip modules score monotonic advances.Together, these terms assess local state plausibility and transition likelihood under the current alignment context.
  • B.5 State Scoring and Stability-Oriented Calibration: Per-frame unary and sample-wide transition canonicalization removes offset, scale, and utterance-level magnitude variation, reducing sensitivity to numerical and implementation differences.The design targets stable Viterbi paths across machines and execution environments.
  • B.5 State Scoring and Stability-Oriented Calibration: Independent learnable gains γu and γτ calibrate unary and transition terms before defining the final transcript-conditioned alignment score.Dual-gamma calibration is more flexible than a single global temperature because occupancy and transition terms require separate scaling.

B.6 Training Objectives · B.7 Inference Procedure

Swan Forced Aligner trains on frame-level state targets derived from word timings, combining local alignment, global path, duration, and monotonicity objectives. At inference, it decodes the same monotonic lattice with Viterbi or posterior-based methods and produces word timings plus confidence diagnostics.

  • B.6 Training Objectives: Word-level time annotations become frame-level supervision over the interleaved topology, assigning lexical frames to word states and other valid frames to position-dependent blank states.These targets define the inference lattice used for alignment.
  • B.6 Training Objectives: The primary objective is cross-entropy alignment loss over valid acoustic frames, optionally weighting frames by annotation confidence.The loss uses predicted state distributions against target states on valid frames.
  • B.6 Training Objectives: A CRF objective over the structured lattice complements cross-entropy by favoring the annotated monotonic path over competing valid paths.This encourages globally consistent alignment rather than only locally accurate frame predictions.
  • B.6 Training Objectives: Duration supervision regularizes occupancy for both word and blank states using absolute-error and log-duration penalties across short and long segments.Predicted occupancies are obtained by summing state posteriors over time.
  • B.6 Training Objectives: A monotonicity regularizer penalizes decreases in expected word index, and the final objective combines the major loss terms with unit weight unless otherwise specified.The regularizer discourages locally inconsistent alignments under ambiguous evidence.
  • B.7 Inference Procedure: Inference computes calibrated unary and transition scores on the training lattice, with default Viterbi decoding recovering word start and end times from assigned frame ranges.Training and decoding use the same monotonic topology.
  • B.7 Inference Procedure: Posterior-based decoding uses forward–backward state posteriors and can be more robust under ambiguous local evidence by incorporating path uncertainty.After decoding, emission-state probabilities averaged over each word span provide confidence signals for debugging and alignment-error analysis.

C Experiments · C.1 Experimental Setup

The experiments evaluate Swan Forced Aligner on large-scale Chinese-English speech data using human-timestamped benchmarks, standardized timestamp metrics, and comparisons with mainstream forced aligners. The setup specifies the model architecture, training configuration, and evaluation protocol.

  • C.1 Experimental Setup: 80K hours of Chinese-English speech spanning audiobooks, podcasts, conversations, meetings, and livestreams train the aligner, with pseudo-timestamps from Montreal Forced Aligner.Evaluation uses the human-timestamped Chinese subset of GTSinger-Speech and Librispeech-Alignment.
  • C.1 Experimental Setup: The aligner uses WavLM, a 4-layer bidirectional Transformer auxiliary encoder, and a 16-layer bidirectional Transformer text encoder.Both Transformers use hidden size 512 and 8 attention heads; the model has about 400M parameters.
  • C.1 Experimental Setup: Training uses 24 A100 GPUs, a 4-hour batch size, 80K steps, AdamW, and a 1.0e-5 learning rate.The supplied implementation passage also specifies β = (0.
  • C.1 Experimental Setup: Timestamp prediction is evaluated with accumulated averaging shift (AAS), where lower values indicate more accurate timestamp prediction.AAS measures average boundary deviation across all evaluated word slots.
  • C.1 Experimental Setup: The evaluation protocol defines AAS over N evaluated word slots and predicted boundary pairs si = (t(i).The supplied mathematical-definition passages are truncated, so only the visible notation is reported.
  • C.1 Experimental Setup: Five mainstream forced aligners serve as baselines, including Monotonic-Aligner, NeMo Forced Aligner, WhisperX, and Qwen3 Forced Aligner.Monotonic-Aligner is described as Chinese-only and Paraformer-based; NeMo uses CTC-based ASR models, WhisperX combines VAD with forced phoneme alignment, and Qwen3 uses parallel slot filling.
  • C.1 Experimental Setup: The baseline evaluation uses official checkpoints or inference procedures for language-specific alignment where specified.NeMo’s official checkpoint is used for English, while WhisperX uses different Chinese and English checkpoints following its official inference script.

C.2 Experimental Results

Swan Forced Aligner achieves the best open-source AAS on Chinese and LibriSpeech-Clean benchmarks, while remaining competitive on LibriSpeech-Others against proprietary and open-source systems.

  • Experimental Results: Table 3 compares forced aligners using AAS (ms)↓ on Chinese and English test datasets.Bold marks the best results, underlining marks the second best, and an asterisk denotes checkpoints unavailable at evaluation time.
  • Experimental Results: Swan Forced Aligner gives the best open-source AAS on the Chinese and LibriSpeech-Clean benchmarks.
  • Experimental Results: 0.18 ms is the gap between Swan Forced Aligner and Qwen3 Forced Aligner on LibriSpeech-Others, while LattifAI Aligner is about 10 ms ahead.LattifAI Aligner is the best proprietary system in this comparison.
Loading 2605.30993v1…