Comprehensive Benchmarking of Long-Form Speech Generation in Diverse Scenarios

Motivation

• Existing tests often cover limited domains or single-speaker settings, leaving models’ performance in complex long-form scenarios underexplored.Downstream applications involve multi-speaker interactions and rich semantic contexts.Source: §1 · p. 2
• Long-form evaluation lacks scalable, accurate measures: WER is saturated and poorly aligned with perception, listening tests are costly, and MLLM judgments can miss consistency.MLLM evaluators typically provide coarse comparative judgments rather than quantitative metrics.Source: §1 · p. 2

Benchmark scope

• SwanBench-Speech covers long-form speech and dialog generation with 1,101 samples across 17 scenarios spanning acoustics, semantics, and expressiveness.The benchmark targets broad coverage of long-form TTS applications.Source: §1 · p. 2

Evaluation protocol

• Its hierarchical automatic protocol adds Acoustic Consistency, Prosodic Coherence, and Expressive Hierarchy to Fidelity and Accuracy, quantifying temporal stability and expressive dynamics.User studies validate the metrics as a scalable proxy for perception.Source: §1 · p. 2

Results

• Current models rival human recordings in fidelity and accuracy but lag in reverb consistency, prosodic coherence, and expressive hierarchy, with performance worsening in highly expressive scenarios.The results point to persistent challenges in modeling long-term dependencies and dynamic stylistic variations.Source: §1 · p. 2

2 Related Work

• Long-form TTS research tackles prosodic coherence with joint style modeling and cross-sentence memory, and sequence efficiency with quantization, low-framerate tokenization, and state space models.Speaker-transition approaches have combined autoregressive and non-autoregressive components; newer non-autoregressive methods use flow matching, while autoregressive models use speaker tokens for long-context dialogue.Source: §2 · pp. 2–3
• Existing metrics remain inadequate for prosodic coherence, emotional richness, and transition quality, motivating SwanBench-Speech’s targeted test cases and human-aligned evaluation metrics.The framework is designed to quantify these properties.Source: §2 · p. 3

Evaluation for Speech Generation Models

• Current speech-generation evaluation is limited by near-saturated objective metrics, sentence-level benchmarks, narrow long-text scenario coverage, and subjective protocols that lack scalability and standardization.SwanBench-Speech jointly covers long-form speech and dialog generation across 17 scenarios with comprehensive automatic metrics aligned with human judgments.Source: Evaluation for Speech Generation Models · p. 3

3.1 Overview

• The benchmark’s 17 scenarios span six acoustic, five semantic, and six expressive scenarios, covering diverse downstream speech applications.Acoustic scenarios target timbre, reverberation, and sound fidelity; semantic scenarios target content accuracy and prosodic coherence; expressive scenarios target sentence-level richness and paragraph-level dynamics.Source: §3.1 · p. 3

3.2 Data Collection

• Samples come from online text corpora, online audio media—the main source—and GPT-5 generation, with human checks applied to the curated materials.Audio is denoised, filtered using DNS-MOS, diarized, and transcribed; online text is cleaned and proofread, while generated cases are checked by annotators.Source: §3.2 · pp. 3–4

3.3 Data Refinement

• Refinement combines semantic deduplication, content-quality and privacy screening, and manual review to yield 1,101 samples across three challenges and 17 scenarios.GPT-5 supports topic-based deduplication and quality screening; DeepSeek V3.2 with a chain-of-thought procedure detects potential leaks and social or ethical risks.Source: §3.3 · p. 4

3.4 Evaluation Metrics

• The seven metrics include Timbre Consistency, Reverb Consistency, Sound Fidelity, Content Accuracy, and Prosodic Coherence, assessing acoustic stability, clarity, lexical correctness, and delivery naturalness.Timbre consistency averages pairwise speaker-embedding similarity across windows; reverb consistency uses the standard deviation of windowed SRMR scores, and sound fidelity uses reference-free SQUIM-PESQ.Source: §3.4 · pp. 4–5
• The remaining metrics are Expressive Richness, scored over 10-second segments, and Expressive Hierarchy, scored on the full utterance to preserve narrative flow.Content Accuracy uses ASR-based Word Error Rate (Character Error Rate for Chinese), while Prosodic Coherence uses a SpeechJudge scalar score focused on pauses, speaking rate, and prosodic consistency.Source: §3.4 · p. 5

3.5 Human Perception Alignment Test

• Prosodic-coherence evaluation aligned with human preferences at SRCC 0.82; Gemini 3 Pro aligned at 0.71 for expressive richness and 0.62 for expressive hierarchy.The prosody test used 50 pairs of clips and 10 evaluators; the expressiveness test used 200 clips and 10 evaluators, comparing three MOS prediction networks and six LALMs.Source: §3.5 · p. 5

4.1 Settings

• The settings compare 10 open-source and six closed-source single-speaker systems, plus six open-source and four closed-source dialogue systems.WavLM TDCNN1 supplies speaker embeddings; Paraformer2 and WhisperX perform forced alignment on Chinese and English data, respectively; FunASR Nano3 supports WER computation, and Gemini3-pro with prompt enhancement evaluates expressiveness metrics.Source: §4.1 · p. 6

4.2 Evaluation from Different Perspectives

• Real Speech and Real Dialogue serve as topological upper bounds for audio quality in the per-dimension evaluation summarized in Tables 2 and 3.Scores cover all dimensions under the Section 3.4 evaluation protocol.Source: §4.2 · p. 6
• Fig. 3 visualizes model evaluation results across three core categories and 17 scenarios for long-form speech and dialogue generation.Performance is calculated using the evaluation protocol.Source: §4.2 · p. 6
• Fig. 4 reports results for five representative models tested at increasing input lengths on 100 samples across acoustics, semantics, and expressiveness.The evaluated models are MegaTTS3, F5TTS, Cosyvoice2, SparkTTS, and VibeVoice.Source: §4.2 · p. 6

5.1 Observations

• VibeVoice and SoulX-Podcast are the strongest open-source models, while Minimax-Speech-02-hd and Gemini-2.5-pro-preview-tts lead proprietary models; proprietary systems perform better overall.Even the best systems trail real speech in expressiveness: the closed-source average lags by nearly one MOS point in richness and over half a point in hierarchy; dialog Reverb Consistency is 3.36 vs. 2.73.Source: §5.1 · p. 6, Table 2 · p. 7, Table 3 · p. 8
• Expressiveness scenarios degrade nearly all models’ metrics, especially Expressive Richness, while acoustic and semantic challenges chiefly expose weaknesses in consistency and prosody.Frequent speaker transitions disrupt reverberation unity; linguistic complexity leaves Content Accuracy intact but challenges prosody modeling.Source: §5.1 · p. 7, Figure 3 · p. 8

5.2 Discussions

• NAR models offer robust, efficient long-text synthesis but weaker expressive hierarchy, whereas AR models excel in prosody yet risk error propagation and lower Content Accuracy.SparkTTS content accuracy declines as sequence length increases, while NAR models remain stable; the authors propose moving beyond this binary choice toward a Coarse-to-Fine Architecture (Kharitonov et al., 2023; Ju et al., 2024).Source: §5.2 · pp. 7–8, Figure 4 · p. 8
• The authors attribute long-form failures to fragmented short-form data, acoustically unstable web recordings, and scaling-induced homogenization of expressiveness, advocating data quality and temporal continuity over raw quantity.They recommend curriculum learning from sentence- to paragraph-level training with high-fidelity, long-context recordings to capture long-range dependencies.Source: §5.2 · pp. 8–9

• SwanBench-Speech covers three core long-form generation challenges with 1,101 curated instances across 17 downstream scenarios.Source: §6 · p. 9
• Its disentangled, human-aligned evaluation protocol enables precise automatic assessment through seven complementary metric dimensions.Source: §6 · p. 9
• Benchmarking over 20 models, the study analyzes current capabilities and limitations across model architectures, training data, and strategy.The authors envision SwanBench-Speech as a standardized testbed to support future research and more robust, immersive long-form speech synthesis.Source: §6 · p. 9

• The benchmark covers only Chinese and English, leaving low-resource languages and diverse dialects or accents underexplored.Source: Limitations · p. 9
• Its semantic investigation remains preliminary: metrics prioritize acoustic coherence, without a robust automated framework for assessing emotionally and stylistically grounded transitions in long-form text.Source: Limitations · p. 9
• Prompt speech comes from only 20 speakers in open-source datasets, potentially introducing evaluation bias; the authors encourage additional data to assess model generalization more comprehensively.Source: Limitations · p. 9

Ethical considerations

• Users must respect voice-actor rights and avoid using the benchmark to generate disinformation; the planned release includes usage guidelines and a non-commercial license.Prompt audio must not come from unverified or restricted sources, and additional voice profiles must follow their asset licenses.Source: Ethical considerations · p. 9, §A.4 · p. 18

A.1 Explanation of Scenarios

• The benchmark organizes 1,101 audio samples across 17 downstream scenarios around acoustics, semantics, and expressiveness.Examples include audiobooks, podcasts, talk shows, and news broadcasting.Source: §A.1 · p. 15

Scenarios for Acoustics Challenges

• Acoustic scenarios test audio fidelity, stable speaker timbre and transitions, and a consistent acoustic environment, including across multi-speaker dialogue.The benchmark selects six acoustic scenarios, including customer service, audiobooks, podcasts, chat, debate, and interview settings.Source: Scenarios for Acoustics Challenges · pp. 15–16

Scenarios for Semantics Challenges

• Semantic evaluation targets content accuracy and prosodic coherence across five scenarios, testing robustness to text errors and natural paragraph-level delivery.News and popular science emphasize correctness; lessons, seminars, and presentations also test stress, cadence, and prosodic variation.Source: Scenarios for Semantics Challenges · p. 16

Scenarios for Expressiveness Challenges

• Expressiveness is divided into richness and hierarchy, assessed in six highly expressive scenarios involving sustained emotion and dynamic delivery.Sportcasts and live streams emphasize sustained intensity, while speech, hosting, talk shows, and drama require expressive control aligned with a narrative arc.Source: Scenarios for Expressiveness Challenges · p. 16

A.2 Details of Data Collection

• Data collection combines cleaned, human-annotated online text and audio with scenario-specific LLM-generated test cases that annotators proofread.Audio is denoised, filtered at a DNS-MOS threshold of 3.5, diarized, transcribed, and checked against ground truth; three undergraduate annotators double-check samples.Source: Online Text Corpora · p. 16, Online Audio Media · p. 17, LLM Generation · p. 17

A.3 Details of Data Refinement

• Topic-level deduplication uses GPT-5-extracted text encoded with Sentence-BERT, removes samples above cosine similarity 0.8, and replenishes the set; GPT-5 quality ratings retain scores above 2.Quality ratings assess textual expressiveness and content consistency on a 1-to-5 scale.Source: Semantic De-duplication · p. 17, Quality Evaluation · p. 17

Privacy and Ethical Filtering

• Privacy and ethical filtering anonymizes private individuals and removes toxic samples, while human reviewers replace placeholders and purge residual errors.The appendix also outlines dataset construction, evaluation protocols, validation and user studies, experimental settings, ablations, further results, limitations, and potential social impact.Source: Privacy and Ethical Filtering · pp. 17–18, Manual Review · p. 18

B.1 Categorical Statistics

• 49.3% of samples are Chinese and 50.7% English; the 1,101 samples span five dimensions, including language and single-, dual-, or multi-speaker configuration.The benchmark focuses on Chinese and English to include more baseline models and validate SwanBench-Speech.Source: §B.1 · pp. 18–19
• 101 samples involve three or four speakers, complementing the dataset’s primary focus on single-speaker long-form speech and dual-speaker dialogue.Source: §B.1 · p. 19
• Acoustics is the largest of the three core challenges at 34.5%; the dataset covers 17 downstream scenarios and varied content topics to minimize evaluation bias.Source: §B.1 · p. 19

B.2 Distributional Statistics

• 271.8 Chinese characters and 174.6 English words are the mean text lengths; both languages’ lengths are approximately normal and concentrate within [80, 500].Lengths count Chinese characters or English words, excluding punctuation and other non-phonetic elements.Source: §B.2 · p. 19
• Text length is measured in Chinese characters or English words, excluding punctuation and other non-phonetic elements.Source: §B.2 · p. 19
• Over 100 words, most models show performance degradation in Timbre Consistency, Reverb Consistency, and Expressive Hierarchy, revealing long-term dependency issues.This finding is reported from generated-length experiments in Section 4.2 and Appendix F.3.Source: §B.2 · p. 19
• 200 to 400 words were selected for minute-level evaluation; Guo et al., 2024b, Zhang et al., 2025d, and Xie et al., 2025a also indicate this duration can reveal long-term dependencies.The choice reflects the importance of minute-level synthesis for most extended-speech applications, while acknowledging that audiobooks may require over 10 minutes or an hour.Source: §B.2 · p. 19

C.1 Timbre Consistency; C.2 Reverb Consistency

• For a single-speaker long-form sample, timbre consistency is the average pairwise cosine similarity among WavLM TDCNN speaker embeddings extracted using 3-second windows with a 2-second stride.Source: §C.1 · p. 20

C.3 Sound Fidelity; C.4 Content Accuarcy

• SQUIM-PESQ provides reference-free sound-fidelity assessment, while content accuracy uses CER for Chinese and WER for English transcripts from FunASR-Nano.Text normalization removes punctuation, standardizes whitespace, converts Traditional Chinese to Simplified, and filters non-ASCII characters in English before calculating error rates.Source: §C.3 · p. 21, §C.4 · pp. 21–22

C.5 Prosodic Coherence

• SpeechJudge rates Prosodic Coherence & Flow, Rhythmic Hierarchy & Layering, and Overall Naturalness on a 1.0–5.0 scale.Ten independent evaluations are averaged for each generated sample to reduce variance in LALM judgments.Source: §C.5 · p. 22

C.6 Expressive Richness; C.7 Expressive Hierarchy

• Expressive Richness averages LALM expressiveness scores across non-overlapping 10-second chunks, while Expressive Hierarchy uses holistic long-context assessment.The hierarchy assessment considers Emotional Variation, Vocal Dynamics, and Scene Appropriateness; chunking richness evaluation helps isolate intrinsic expressiveness from inter-chunk inconsistencies.Source: §C.6 · pp. 22–23, §C.7 · p. 23

D User Study

• The subjective evaluation uses MOS ratings from 10 experienced experts, balanced by gender, across audio engineering, live streaming, and academic backgrounds.Participants receive $1.00 per evaluation instance, and the user study costs $2,000 in total.Source: D User Study · p. 23

D.1 Validation of Timbre Consistency

• SRCC=0.75, PLCC=0.77, and KRCC=0.59 show that the objective Timbre Consistency score aligns closely with human MOS.The study randomly selected 50 test samples; scores below 0.85 indicate significant drift, 0.85–0.90 is generally acceptable, and scores below 0.93 indicate maintenance comparable to ground truth. Global averaging may miss periodic timbre fluctuations.Source: §D.1 · p. 23

D.2 Validation of Sound Fidelity

• SRCC of 0.72, PLCC of 0.47, and KRCC of 0.53 indicate that SQUIM-PESQ aligns with human ratings of clarity and fidelity.The validation used 50 test samples to assess generalization to Chinese and long-form audio, with listeners instructed to ignore prosody and expressiveness.Source: §D.2 · p. 23

D.3 Validation of Prosodic Coherence

• A prosody-score difference greater than 1 indicates a substantial, perceptually obvious gap in prosodic quality between audio samples, while scores of at least 4.5 are considered virtually indistinguishable from ground-truth recordings.A score of 4 demonstrates competent basic prosody and rhythmic structure.Source: §D.3 · p. 24

D.4 Validation of Expressiveness

• The subjective evaluation uses 200 samples across models and tasks, rated by listeners against the same prompt criteria used for LALMs.The benchmark also compares the test set with four MOS predictors and eight flagship LALMs; only a subset of LALMs is used for Expressive Hierarchy evaluation due to context limits.Source: §D.4 · p. 24
• Gemini3-Pro outperforms other models on both expressive metrics, while Qwen3-Omni-Flash and Qwen3-Omni-Instruct outperform GPT-4o; traditional MOS predictors correlate poorly with human expressiveness judgments.The authors suggest standard MOS training datasets may lack a specific focus on expressive qualities.Source: §D.4 · p. 24, Table 5 · p. 24
• Across five independent scoring iterations, Gemini 3 Pro gives inconsistent scores for only 11 samples, and high inter-rater correlation supports the evaluation protocol’s reliability.The authors consequently adopt single-pass evaluation for this metric.Source: §D.4 · p. 24, Table 6 · p. 25

E.1 Computational Resources and Environments

• Open-source inference and evaluation use a server with eight NVIDIA GeForce RTX 4090 GPUs, an Intel Xeon Gold 6530 CPU, Ubuntu 22.04, Python 3.10, Py-Torch 2.8.0, Torchaudio 2.8.0, and Transformers 4.57.3.Model inference follows the environment specifications in each model’s official repository.Source: §E.1 · p. 25

E.2 Selected Voice

• Open-source models are evaluated with 25 reference audio prompts spanning over 20 timbres, and results report each model’s best-performing voice; closed-source models use selected official voices.The open-source prompts span multiple datasets and cover language, gender, and age; the authors acknowledge timbre coverage limitations.Source: §E.2 · p. 25, Table 7 · p. 25

E.3 Synthesis Strategy

• Open-source models follow repository defaults, with specified adjustments for MegaTTS3, IndexTTS2, and CosyVoice3; closed-source models use default synthesis without manual attribute changes.All open-source models use zero-shot long-form and dialogue generation, and all generated audio is resampled to 24kHz.Source: §E.3 · pp. 25–26

F.1 Inference Speed

• Non-autoregressive models show a significant generation-speed advantage over autoregressive models under the Real Time Factor (RTF) metric.The comparison concerns open-source models; the paper attributes the advantage to parallel decoding.Source: Table 9 · p. 26, §F.1 · p. 26

F.2 Ablation on Window Size

• At windows ≤2s, real recordings show lower Timbre Consistency than CosyVoice3; windows ≥4s reduce the real–synthetic discrepancy, while stride has no significant impact.The study therefore selects a 3s window and 2s stride to improve efficiency and reduce computational overhead.Source: §F.2 · pp. 26–27
• For Reverb Consistency, a 1s window is differentiating but unstable, while windows ≥4s reduce inter-model differences; model rankings remain consistent.VibeVoice has an excessively high standard deviation relative to its mean reverb score of 9.25 at 1s; the selected setting is 3s with a 2s stride.Source: §F.2 · p. 27

F.3 Ablation on Generated Length

• Timbre Similarity is tracked by averaging cosine similarity between WavLM TDCNN24 embeddings of generated 3s segments and the reference embedding.Segments use a 2s stride; the methodology follows Huynh-Nguyen et al. (2025).Source: §F.3 · p. 27
• Timbre Similarity and Timbre Consistency remain comparatively stable as generation length grows, highlighting the effectiveness of “in-context learning” in preserving speaker identity (Du et al., 2024; Jiang et al., 2025).Most models also demonstrate robust Content Accuracy, except SparkTTS, which the paper attributes to modern TTS architectures’ text understanding and alignment capabilities.Source: §F.3 · p. 27
• As generation length increases, nearly all metrics decay, most sharply in Reverb Consistency, Prosodic Coherence, and Expressive Hierarchy.These findings suggest that current models struggle to maintain acoustic-field stability and capture long-term dependencies in long-form settings.Source: §F.3 · p. 27, Figure 10 · p. 28

F.4 Multi-Speaker Dialogue Generation

• The benchmark includes 101 test cases for 3- and 4-speaker dialogue and evaluates ElevenLabs Multilingual V2, Gemini-2.5-pro-preview-tts, and OpenAI-tts-1-hd.The passage refers to Table 13 for results but provides no result values.Source: §F.4 · p. 28

G.1 Detailed analysis on each metric

• Real and synthetic speech show a small Timbre Consistency gap, while dialogue models show weaker Reverb Consistency than real data.Timbre Consistency is 0.96 vs. 0.93 for single speakers and 0.95 vs. 0.92 for two speakers; open-source models have greater standard deviation, and dialogue models show greater variance, likely reflecting speaker transitions. Single-speaker Reverb Consistency is comparable to human recordings except for CosyVoice and ElevenLabs, but dialogue models show sound-field and volume inconsistencies; the reported open and closed averages are 3.45 and 3.36.Source: §G.1 · p. 28
• Generated speech aligns closely with real data in Sound Fidelity, while Content Accuracy remains a challenge for some long-form autoregressive models and synthetic speech trails real data in Prosodic Coherence.FishSpeech and ElevenLabs exceed the real-data mean in Sound Fidelity, which the paper attributes partly to noise in web-crawled recordings. CosyVoice2 and MegaTTS3 perform exceptionally with chunk-based in-context learning, whereas SparkTTS accuracy declines as text length increases; closed-source models outperform open-source models in Prosodic Coherence.Source: §G.1 · pp. 28–29
• Real data exceeds open-source models by approximately 1.5 points and closed-source models by nearly 1.0 point in Expressive Richness.Models also underperform in high-expressiveness settings. For Expressive Hierarchy, real data outperforms synthetic speech, closed-source models surpass open-source ones, and single-speaker scores are lower than Expressive Richness; dialogue models generally score higher, possibly because their semantic logic supplies contextual cues for learning hierarchy.Source: §G.1 · p. 29

G.2 Analysis based on the scenarios

• Most metrics degrade in high-expressiveness scenarios, with sportscast, host, and talk-show showing the steepest declines.The analysis attributes the declines to models’ limited ability to model highly dynamic prosody and intense emotional variation.Source: §G.2 · p. 29, Figure 11 · p. 31

G.3 Analysis based on the Languages

• Language significantly affects many models’ performance: ElevenLabs Multilingual V2 scores 1.79 vs. 2.87 in Expressive Richness across Chinese and English, while Seed-TTS-Podcast scores 4.19 vs. 3.49.Gemini-2.5-pro-preview-tts is reported to perform exceptionally in prosody and expressiveness while maintaining balance across both languages.Source: §G.3 · p. 30, Table 14 · p. 32, Table 15 · p. 33

H Future Works

• Future work should develop reproducible open-source expressiveness evaluators and extend the English-and-Chinese-only dataset to other languages, especially low-resource ones.Expressiveness evaluation currently relies on closed-source models such as Gemini 3 Pro, whose API updates pose reproducibility risks; the paper proposes distillation using human assessments and closed-source model outputs (Ji et al., 2024a).Source: H Future Works · p. 30
• The benchmark’s over 20 reference voices may not capture enough voice diversity, and its zero-shot focus leaves long-context instruction-following evaluation as future work.Prior work (Manku et al., 2025) notes that expressiveness and prosody are sensitive to the reference voice; the paper also identifies evaluating InstructTTS systems as an avenue for future research (Huang et al., 2025; Wang et al., 2025; Zhou et al., 2025b; Xu et al., 2025b).Source: H Future Works · p. 30

I Social Impacts

• The paper identifies misuse risks and describes ethical review, anonymization, PII checks, and a CC BY-NC-SA 4.0 license as safeguards for benchmark use.It expresses the hope that speech-generation progress benefits society through responsible and ethical deployment.Source: I Social Impacts · p. 30