PaperlayerView exact revision on arXiv

Source-linked AI summary

DRAMA-Eval: A Benchmark for Context-Aware, Multi-Turn, Multi-Speaker Speech Generation

Jingbin Hu, Zihang Yang, Xiaoge Peng, Haoyu Zhang, Ziyu Zhang, Qirui Zhan, Huakang Chen, Zhao Guo, Wenjie Tian, Kangxiang Xia, Wenkang Hou, Ruonan You, Zihan Liu, Chenyi Liao, Hongyu Lyu, Zihao Bai, Jinrui Liang, Shengming Lin, Yunxiang Chen, Houdun Liu, Su Feng, Bengu Wu, Liumeng Xue, Lei Xie

arXiv:2610.31796v1eess.AScs.SD

TL;DR

Existing speech-generation benchmarks often assess instruction following, multi-turn generation, and speaker consistency separately, despite growing demand for context-aware script-driven synthesis. DRAMA-Eval evaluates these abilities together using structured scripts and real-world reference audio; its experiments reveal complementary weaknesses across expressive control, speaker consistency, and coherent interaction.

  • Problem

    Speech-generation evaluation has not kept pace with context-aware, multi-turn, multi-speaker systems, and existing assessments often separate contextual instruction following, turn-taking, and extended speaker consistency.

    Source: §1 · pp. 1–2

  • Method

    DRAMA-Eval evaluates complete scene audio from structured scripts through objective fidelity and speaker-consistency measures, local instruction and expression judgments, and global interaction judgments.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Experiments across four generation paradigms reveal complementary limitations in expressive control, voice consistency, and coherent interaction; strong utterance-level instruction execution does not necessarily yield a coherent scene.

    Source: §4.3 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    Providing global context improves all three scene-level evaluation dimensions in both languages, supporting its value for speech perceived as appropriate to the overall scene.

    Source: §4.5 · p. 8

    The benchmark covers only Chinese and English, while its automated evaluation relies on a proprietary audio-language model whose biases and updates may affect reliability and reproducibility.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent advances in speech generation are shifting toward context-aware, multi-turn, and multi-speaker synthesis. While these capabilities are in high demand for applications such as audiobook and audio drama production, evaluation has not kept pace with these advances. Beyond traditional zero-shot Text-to-Speech (TTS), modern systems in this domain primarily fall into three categories: instruction-controllable generation models, multi-turn dialogue models, and the audio branches of audio-visual generation models. However, existing benchmarks evaluate these paradigms in isolation, failing to provide a unified assessment of context modeling, instruction following, and multi-speaker generation. Meanwhile, recent progress in audio understanding models has made it feasible to evaluate such complex scenarios using LLM-as-a-Judge approaches. To bridge this gap, we introduce DRAMA-Eval, a comprehensive benchmark for evaluating such systems. DRAMA-Eval utilizes structured script contexts and offers a holistic evaluation framework covering objective, local, and global aspects of generation quality, enabling the assessment of acoustic fidelity, sentence-level execution, and long-range coherence. We benchmark leading open-source and commercial models on a human?verified dataset. The results reveal critical weaknesses in long-range speaker consistency and implicit instruction following that require deep contextual understanding. To facilitate reproducible research, we release a 12-hour bilingual (Chinese-English) test set of manually corrected real-world scripts, accompanied by ground-truth audio, evaluation code, and benchmark results.

1 Introduction

Speech generation is moving toward script-driven, multi-turn, multi-speaker use, but evaluation often misses contextual instruction following, turn-taking, and long-range speaker consistency. DRAMA-Eval combines these evaluation targets in a benchmark built around structured scripts and real-world audio.

1 Introduction

  • Existing assessments often isolate instruction following, short-clip expressiveness, or multi-turn identity, leaving contextual execution, natural turn-taking, and extended speaker consistency undermeasured.Instruction models commonly target isolated utterances, while dialogue models often use transcripts and speaker IDs rather than structured script context.Source: §1 · p. 2
  • DRAMA-Eval jointly evaluates objective fidelity, local execution, and global consistency for multi-turn, multi-speaker speech generated from structured scripts.Its protocol also uses A/B preference judgments against ground-truth audio and reports human consistency studies.Source: §1 · p. 2
  • The released benchmark includes 12 hours of Chinese-English manually corrected real-world scripts paired with ground-truth audio.Sources include films, TV dramas, podcasts, and interviews.Source: §1 · p. 2
  • Benchmarking leading open-source and commercial systems reveals persistent weaknesses in long-range consistency and context-driven implicit instruction following.These capabilities require deep contextual understanding.Source: §1 · p. 2

2 Related Work

Speech-generation evaluation has expanded from acoustic quality toward instruction control, contextual expression, long-form consistency, and multi-speaker interaction. LLM-based judges offer a scalable approach, with judge models and protocols tailored to different generated-audio criteria.

2.1 History of Speech Generation Evaluation

2.2 LLM-as-a-Judge for Speech Generation Evaluation

  • LLM-as-a-Judge benchmarks use varied judges and criteria, including Gemini for instruction or reference comparisons and a trained Qwen3-Omni-based judge for fine-grained audio alignment.These studies illustrate scalable model-based judging tailored to different aspects of generated audio.Source: §2.2 · p. 3

3 DRAMA-Eval

DRAMA-Eval tests whether speech-generation systems can jointly maintain character identity, follow expressive instructions, and sustain coherent interactions across structured, multi-turn scenes. It evaluates complete-scene audio at objective, local, and global levels using a bilingual real-world dataset with human-reviewed scripts and reference audio.

3.1 Task Definition and Problem Formulation

  • The task asks whether systems can jointly maintain speaker identity, execute expressive instructions, and sustain coherent interactions across extended scenes.The benchmark conditions generation on structured scripts containing scene context, character descriptions, speaker assignments, target text, and local instructions.Source: §3.1 · pp. 3–4

3.2 DRAMA-Eval: Methods

  • Evaluation targets complete-scene audio and covers objective textual fidelity and speaker consistency, local delivery, and global interaction and scene quality.The target may be generated natively as dialogue or assembled from utterance-level outputs, allowing assessment of cross-turn discontinuities.Source: §3.2 · p. 5
  • Generated speech is compared with corresponding real-world audio in blinded A/B judgments, with ties counted at half weight and reliability checked against human preferences.The protocol reverses A/B order, requires preference in both presentations, and averages results across independent evaluation runs.Source: §3.2 · p. 5

3.3 DRAMA-Eval: Dataset

  • Scripts are built from diverse Chinese- and English-language real-world speech through segment annotation, structured rewriting, and human review; source audio serves as the evaluation reference.The scripts organize global scene context, character profiles, and local delivery instructions.Source: §3.3 · p. 5
  • 1,038 cases (537 Chinese, 501 English) span 12.50 source-audio hours and 7,530 turns; 65.6% are multi-speaker and 68.0% contain 6–10 turns.Cases cover one to four speakers and four evaluation categories: Normal, Escalation, Contrast, and Rich Emotions.Source: §3.3 · pp. 5–6

4 Experiments

The experiments compare four speech-generation configurations through objective measures, local and scene-level judgments, human agreement, and a global-context ablation. Results show complementary gaps between textual fidelity, voice consistency, expressive execution, and coherent scene generation.

4.1 Inference Protocols

  • The four inference protocols use systems' supported conditioning interfaces, so cross-category comparisons reflect generation configurations rather than identical contextual inputs.Context-aware models receive the structured script and generate a whole scene; the other protocols use differing combinations of reference audio, speaker assignments, and utterance-level instructions.Source: §4.1 · pp. 6–7

4.2 Objective Evaluation

  • SeedAudio has the lowest Chinese CER at 2.58, and FireRedTTS3 has the lowest English WER at 3.97; strong textual fidelity spans categories.Dialogue TTS systems also show strong speaker consistency: MOSS-TTSD scores approximately 0.78/0.77 on Chinese/English and SoulX-Podcast approximately 0.78/0.76.Source: §4.2 · p. 7

4.3 LALM-Based Evaluation

  • The LLM judge compares generated audio with real-world references on six local and global dimensions, using reversed A/B presentations and tie-adjusted win rates.The six dimensions cover instruction following, prosody and fluency, local expressiveness, interaction naturalness, overall fluency, and scene immersion.Source: §4.3 · p. 7
  • SeedAudio leads several local and scene-level dimensions, while strong speaker consistency or local control does not guarantee expressive delivery or coherent scenes.SeedAudio scores 62.2/56.5 on B1 and 64.2/57.0 on B3 for Chinese/English, and 67.4/73.2/75.7 and 56.8/54.5/55.7 on C1/C2/C3; Qwen3-TTS has Chinese C2/C3 scores of 28.2/27.7 despite competitive local scores, while MOSS-TTSD has Chinese B3 of 35.9 in its reference-only configuration.Source: §4.3 · p. 8

4.4 Agreement with Human Judgments

  • Human agreement with the automated judge ranges from 65.23%–73.05% at annotator level and 65.91%–81.58% at consensus level; consensus agreement is higher across all six dimensions.Interaction naturalness (C1) has the highest agreement value.Source: §4.4 · p. 8, Table 4 · p. 8

4.5 Global Context Ablation

  • Providing global context improves all three scene-level dimensions in both languages in the SeedAudio ablation.The comparisons use generated scenes against real-world audio.Source: §4.5 · p. 8

5 Conclusion

DRAMA-Eval combines bilingual, real-world-script evaluation with objective, local, and global quality assessment. Experiments identify complementary generation weaknesses, support the judging protocol through human agreement analysis, and underscore the importance of global scene information.

5 Conclusion

  • DRAMA-Eval evaluates context-aware, multi-speaker speech generation from structured scripts grounded in real-world audio.Its framework jointly assesses textual fidelity, speaker consistency, local instruction execution, and scene-level interaction quality.Source: §5 · pp. 8–9
  • Table 5 reports a SeedAudio ablation of global context using C1–C3.The supplied conclusion identifies global scene information as important but provides no table values.Source: Table 5 · p. 9, §5 · p. 9
  • Experiments across four generation paradigms reveal complementary limitations in expressive control, voice consistency, and coherent interaction.Human agreement analysis supports the automated judging protocol, while context ablations highlight the importance of global scene information.Source: §5 · p. 9

6 Limitations

The benchmark's current coverage and evaluation setup bound how broadly its results can be generalized and interpreted.

6 Limitations

  • The dataset covers Chinese and English only, limiting generalization to other languages and cultural contexts.Its scenes do not capture extended narrative generation.Source: §6 · p. 9
  • The automated judge is proprietary, and its biases or updates may affect reliability and reproducibility despite human validation.Source: §6 · p. 9
  • The dataset is imbalanced across emotions and speaker configurations, and evaluation omits background ambience, music, and sound effects.Models also receive different conditioning inputs according to their supported interfaces, which matters when interpreting cross-category comparisons.Source: §6 · p. 9

A Evaluation Prompts and Examples

The appendix specifies separate pairwise judging prompts for local utterance qualities and complete-scene qualities, then illustrates the structured script format used to provide context.

A.1 Local LALM Evaluation Prompt

  • The local prompt compares two clips turn by turn, with separate judgments for instruction following, prosody and fluency, and expressiveness.The workflow infers expected delivery from script context before comparing the audio.Source: §A.1 · p. 15
  • The local prompt calls for ties when differences are negligible and requires a structured output with a winner of A, B, or Tie.Its examples distinguish synthesis artifacts and rhythm from expressive vocal performance.Source: §A.1 · pp. 15–16
  • B1 judges fidelity to delivery instructions, B2 physical acoustic quality, and B3 acting depth and dynamic range.The prompt directs judges to use context rather than spoken-text semantics when they conflict, and not to reward or penalize background music or natural noise.Source: §A.1 · pp. 15–16

A.2 Global LALM Evaluation Prompt

  • The global prompt compares complete scene audio on interaction naturalness, overall flow, and scene immersion, with the script supplied for context.It asks judges to listen to the whole clips and keep the three global metrics separate.Source: §A.2 · pp. 16–17
  • Global judgments distinguish conversational feedback, acoustic continuity, and theatrical immersion; the prompt excludes background music as a basis for rewarding immersion.It defines C1 as inapplicable for a single-speaker script, where the judge must report that it was skipped and return a tie.Source: §A.2 · pp. 17–18

A.3 Script Example

  • The script example organizes a scene into background context, character profiles, and utterances with speaker identities, text, timing, and local stage directions.The example depicts a Chinese police-meeting scene with four turns.Source: §A.3 · pp. 18–20
Loading 2610.31796v1…