PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Benchmark: Conditioning, Domains and Failure Modes of Qwen3-TTS

Séverin Baroudi, Yanis Labrak, Petr Motlicek, Hervé Bredin, Ricard Marxer

arXiv:2610.23678v1cs.SDcs.CL

TL;DR

The paper examines how Qwen3-TTS’s two cloning interfaces perform across domains and where they fail, addressing gaps in guidance beyond read-speech benchmarks. It compares the interfaces and other systems using listening tests, acoustic measures, an LLM judge, and a spoofing countermeasure. Neither interface dominates, domain explains more variance than system choice, and the off-the-shelf detector fails on ATC radio.

  • Problem

    Existing evaluations focus mostly on clean read speech, and no single evaluation family establishes how Qwen3-TTS’s two interfaces behave across domains or which acoustic dimensions reveal clones.

    Source: §1 · p. 1

  • Method

    The study compares Qwen3-TTS’s ICL and X-Vector interfaces with three systems across four speech domains using listening tests, acoustic measures, an LLM judge, and a spoofing countermeasure.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    Neither interface dominates; domain explains four times more variance than system choice, and the off-the-shelf detector fails for every system on ATC radio.

    Source: §3.2 · p. 3, §4 · p. 4

  • Takeaways & Limitations

    Choose the interface by reference quality and task: ICL suits verified transcripts over ten words, while X-Vector suits short, noisy, or untranscribed references and timing-sensitive cases.

    Source: §4 · p. 4

    The four objective dimension scores are equally weighted by deliberate choice rather than calibration, and the paper tests this choice against human judgments.

    Source: §2.3 · p. 2

Abstract

from arXiv · show

Qwen3-TTS reports state-of-the-art zero-shot voice cloning, but its technical report leaves open how the model should be conditioned, how it behaves outside read speech, and how it fails. We characterise its two cloning interfaces, in-context learning (ICL) and X-Vector, against Chatterbox, XTTS-v2 and F5-TTS on read, meeting, emotional and air-traffic-control (ATC) speech, using eighteen objective measures, an LLM judge, a spoofing countermeasure and a listening test. Neither interface dominates: ICL clones timbre more closely from correctly transcribed references, X-Vector clones timing, and ICL degrades sharply when a short reference transcript is wrong. Domain explains four times more variance than system choice. Qwen3-TTS leads on intelligibility and pitch register but trails on prosody, and the objective measures rank systems differently from listeners. An off-the-shelf countermeasure flags its clones no more often than genuine recordings, and on ATC radio it fails for every system. Code and audio are released.

1. INTRODUCTION

The paper addresses gaps in evidence about Qwen3-TTS’s two cloning interfaces, performance beyond clean read speech, and failure modes. It combines several evaluation approaches across four domains to investigate those questions.

1. INTRODUCTION

  • Qwen3-TTS reports state-of-the-art zero-shot cloning and exposes two distinct cloning interfaces, but its technical report does not guide practitioners on interface choice, out-of-domain behavior, or failures.Several open systems report human parity on read-speech benchmarks.Source: §1 · p. 1
  • Existing evaluations focus mostly on clean read speech, and common metrics saturate or fail to identify which acoustic dimensions reveal clones or how quality changes across domains.The cited evaluation families also do not cover Qwen3-TTS or its two interfaces.Source: §1 · p. 1
  • No single evaluation family suffices, motivating the paper’s combination of evidence across domains with different recording conditions, speaking styles, and vocabulary.The motivation also situates cloning evaluation amid concerns about listener detection, voice-phishing, and synthetic-audio disclosure.Source: §1 · p. 1
  • The study compares five systems across four domains using listener judgments, a spoofing countermeasure, interface and failure analyses, acoustic measures, and an LLM judge.The authors release the code and generated audio publicly.Source: §1 · p. 1

2. EXPERIMENTAL SETUP

The experiments compare public, untuned systems across four speech domains and evaluate perceived realism, acoustic similarity, hallucinations, and spoof detection. Measures are selected against human judgments, while the hallucination rule uses timing and the detector remains off the shelf.

2.1. Domains, systems and cloning conditions

  • Four one-hour English test sets cover clean read speech, far-field meetings, acted emotional dialogue, and jargon-dense ATC radio; systems use public checkpoints without fine-tuning.All systems receive the same reference clips; Qwen3-TTS uses sampling, while baselines retain library-default hyperparameters.Source: §2.1 · p. 1

2.2. Perceptual test and LLM judge

  • The study combines listener 2AFC judgments on AMI with Gemini 3.5 Flash judgments across all four corpora, reversing pair order as an order-dependence diagnostic.The listener study uses three entry-level annotators; the LLM’s reported rate uses the first ordering, with the real clip as A.Source: §2.2 · p. 2

2.3. Objective measures

  • Eighteen automatic measures span speaker, prosody, F0, and intelligibility; 13 are retained using Spearman correlation of at least 0.45 with human judgments on a separate development set.The four dimension scores are averaged equally, a deliberate but uncalibrated choice tested against human judgments.Source: §2.3 · p. 2

2.4. Hallucination estimation

  • Hallucinations are flagged from duration rather than lexical content: one threshold targets clips too short for their word count, and another flags clips running past their reference.The shared rule supports cross-system and cross-corpus comparisons despite overestimating absolute hallucination rates.Source: §2.4 · p. 2
  • 94.7% recall and 68.8% F1: the timing-based hallucination rule misses 3 of 57 hallucinations, while 3 of 100 unflagged clips are hallucinated.Flagged rates overestimate the true rate, but applying the same rule across systems and corpora keeps comparisons valid.Source: §2.4 · p. 2

2.5. Spoofing countermeasure

  • AASIST is applied as an off-the-shelf countermeasure, without retraining or threshold recalibration, to 100 shifted pairs per cell after fixed-length 16 kHz mono preprocessing.The study reports clone detection and genuine-recording false-alarm rates.Source: §2.5 · p. 2

3. RESULTS

Across read, meeting, emotional, and ATC speech, Qwen3-TTS is difficult for listeners to distinguish from real speech, but its interfaces trade off timbre, timing, and transcript robustness. Corpus and conditioning failures strongly shape results, while detection and automatic scores do not consistently track human judgments.

3.1. Perceived realism and alternatives

  • On AMI, listeners chose Qwen3-TTS clones as real in 35.6% of X-Vector rounds and 27.0% of ICL rounds, versus at most 9.0% for baselines.Chatterbox is the most robust alternative by objective score, but fooled listeners only 9% of the time.Source: §3.1 · p. 3, Table 1 · p. 3

3.2. Machine detection

  • In seven of eight measured cells, AASIST flagged genuine recordings at least as often as Qwen3-TTS clones; on ATC it flagged 99% of genuine recordings versus 75–95% of clones.XTTS-v2 is the only system detected convincingly across all four corpora (86–91%), despite low listener fooling on AMI (6%).Source: §3.2 · p. 3, Table 2 · p. 3

3.3. In-context learning or X-Vector?

  • ICL better matches timbre from clean references, while X-Vector better matches prosody across corpora; overall scores are nearly tied at 33.4 versus 34.3.ICL hallucinates 4.1% versus 2.5% for X-Vector; the listener difference on AMI was too small for the panel to settle.Source: §3.3 · p. 3

3.4. Domains and failure modes

  • Corpus explains 63% of score variance versus 15% for system choice; ATCO2 is hardest (18.5 mean S), with intelligibility 14.7 versus 55.2 on LibriSpeech.IEMOCAP has the lowest F0 and prosody scores (10.5, 19.2).Source: §3.4 · p. 3, Table 3 · p. 4
  • Under whole-word corruption, hallucinations rise from 0.8% to 21.2% on IEMOCAP and 1.6% to 15.2% on ATCO2; rates fall as transcripts exceed eight words.The distractor is reproduced in 7.0% of utterances and, when leaked, appears in the first third in 96.6% of cases, often as the first word (57%).Source: Figure 1 · p. 3, §3.4 · p. 4

3.5. Acoustic strengths and weaknesses

  • Qwen3-TTS achieves the highest intelligibility on three corpora and highest SF0 on all four, but prosody is its main acoustic weakness.ICL has the best average speaker similarity among five systems, though not in every domain; Chatterbox and XTTS-v2 are closer to real timing on three corpora.Source: §3.5 · p. 4

3.6. Can automatic measures replace listeners?

  • Automatic measures capture some background, breathing, and global-timing cues, but miss emotional intensity; the LLM judge is also unstable, with presentation orders disagreeing on up to 43% of pairs.SQUIM SI-SDR correlates most strongly (0.81), but is reference-free and partly tracks corpus cleanliness rather than clone differences.Source: §3.6 · p. 4
  • Averaged over four corpora, S ranks systems like AMI listeners—Qwen3-TTS (34.3, 33.4), Chatterbox (31.9), XTTS-v2 (28.2), then F5-TTS (21.1)—but matched comparisons diverge.On AMI, S ranks XTTS-v2 first (40.9), while human fooling is 6%; across systems S correlates weakly with fooling (ρ = 0.20).Source: §3.6 · p. 4

4. DISCUSSION AND CONCLUSION

Neither Qwen3-TTS interface dominates: the best choice depends on the reference, while domain explains more variance than system choice. The study finds strengths and weaknesses across systems and measures, and concludes that listener tests in the target domain remain necessary.

  • ICL suits clean, verified references over ten words when timbre matters; X-Vector suits short, noisy, or untranscribed references and cases where timing matters.Neither interface dominates, and the domain explains four times more variance than system choice, making short in-domain validation more useful than switching models.Source: §4 · p. 4
  • Qwen3-TTS leads on intelligibility and pitch register and is hardest to distinguish from real speech, but lags on prosody and per-speaker pitch.Chatterbox is a robust second, while XTTS-v2 should be restricted to read speech.Source: §4 · p. 4
  • Neither objective scores nor an LLM judge reproduces listener rankings, and no tested measure tracks perceived realism closely enough to replace listeners.The paper identifies a small forced-choice test in the target domain as the only arbiter.Source: §4 · p. 4
  • An off-the-shelf countermeasure flags Qwen3-TTS clones no more often than genuine recordings outside read speech and is unusable on ATC radio at any threshold.The paper presents disclosure as more dependable than post-hoc detection and prioritizes channel-matched countermeasures for ATC.Source: §4 · p. 4

6. COMPLIANCE WITH ETHICAL STANDARDS

The study used publicly available or institutionally licensed speech corpora. Adult volunteers consented to the listening test, and their responses were anonymous; no personal data were collected.

  • The speech corpora were openly available or distributed under institutional licence.The named corpora were LibriSpeech, AMI, IEMOCAP, and ATCO2.Source: §6 · p. 5
  • Adult volunteers gave informed consent to participate in the listening test.Source: §6 · p. 5
  • Only anonymous listening-test responses were recorded, and no personal data were collected.Source: §6 · p. 5

Related papers

Loading 2610.23678v1…