PaperlayerView exact revision on arXiv

Source-linked AI summary

Refinement Buys Intelligibility, Search Buys Identity: What Test-Time Compute Buys in Masked-Diffusion TTS

Nityanand Mathur, Hamees Sayed, Ayush Pratap Singh

arXiv:2610.03320v1cs.AI

TL;DR

The paper asks whether model depth and inference-time refinement scale equally across TTS capabilities. It trains a width–depth grid and sweeps refinement at inference, finding that refinement closes more of the reachable intelligibility range than identity range.

  • Problem

    It is unclear whether inference-time refinement helps TTS capabilities equally, limiting when steps can trade off against model depth.

    Source: §1 · pp. 1–2

  • Method

    The paper trains a width–depth grid of masked-diffusion TTS models, sweeps refinement steps at inference, and measures intelligibility and identity against floors.

    Source: §1 · p. 2

  • Results

    86.2% of the reachable intelligibility range versus 46.4% of the reachable identity range is closed by refinement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Intelligibility is bounded by refinement, while identity is bounded by training compute and the codec, so refinement cannot reach those identity limits.

    Source: §4.2 · p. 4

    All models are under-trained, and 62% of the distance from codec round-trip similarity to real-speech similarity is lost in the codec.

    Source: §5 · p. 8

Abstract

from arXiv · show

Diffusion language models for text-to-speech combine two forms of computation: model depth (parameters) and refinement steps (inference budget). We ask whether they scale equally across capabilities. We train 15 masked-diffusion codec TTS models varying depth (19-133M parameters, 3 seeds) on 2,000 hours of speech and sweep refinement steps T in [1,16] at inference, measuring zero-shot synthesis via ASR word error rate (intelligibility) and speaker verification (identity) on 174 held-out speakers. Against measured floors, refinement closes 86.2% of the intelligibility range but only 46.4% of the identity range - a 1.86x asymmetry robust across multiple error metrics. Retraining at 3x and 6x schedule attenuates but does not reverse this gap (1.84 to 1.36 to 1.23x), because intelligibility saturates with steps while identity continues improving. Best-of-K search recovers speaker identity where refinement fails, with 64.6-79.0% win rates across four independent encoders. Depth and steps are not interchangeable: separable B(d)B(T) fits significantly better (Delta AICc=+69.3) than substitution models. Analysis shows 62% of remaining identity deficit lies in the codec, not the generator. We conclude that refinement and depth target different bottlenecks and should be optimized separately.

1 Introduction

The paper asks whether inference-time refinement can substitute for model depth across TTS capabilities, which matters because that trade is available only where refinement helps. It finds that refinement closes more of the reachable intelligibility range than identity range, while depth and steps do not exchange at a fixed rate.

1 Introduction

  • Refinement is rented serial compute chosen at serving, whereas depth is owned serial compute fixed at training.The paper asks whether rented computation helps every capability equally.Source: §1 · pp. 1–2
  • Zero-shot TTS separates compositional intelligibility, measured by word error rate, from prompt-carried speaker identity, measured by SIM-o.Source: §1 · p. 2
  • A width–depth grid swept over refinement steps produces a response surface, testing metric selectivity, depth–step exchange, shape effects, and small-budget prediction.The two fitted forms are separable scaling and substitution of steps for depth through dTκ.Source: §1 · p. 2
  • The fitted step-exponent comparison changes sign under monotone non-affine error transformations, limiting what that exponent alone establishes.Source: §1 · p. 2
  • 86.2% of the reachable intelligibility range versus 46.4% of the reachable identity range is closed by refinement; substitution fits worse than separable scaling.The substitution form has ΔAICc = +69.3 for WER and +71.3 for SIM-o.Source: §1 · p. 2
  • The contribution is a floor-referenced, model-free measure of refinement’s per-capability gains, alongside scaling surfaces and an axis-by-axis account of identity.The released study reports 75 points per metric from 45 trained models; its identity analysis includes selection and codec-ceiling comparisons.Source: §1 · p. 2

2 Related work

Prior masked-generative TTS work generally fixes refinement budgets or tests only a few settings, while related research studies scaling, depth, or test-time search in other ways. This paper frames depth and refinement as separate compute levers and asks how they affect intelligibility and identity.

2 Related work

  • Masked-generative TTS recipes fix step budgets or test a few settings; MaskGCT reports WER improving more than speaker similarity in its step sweep.Source: §2 · p. 2
  • Verifier-guided search is studied in reasoning and image diffusion; in TTS, speaker-verifier beam search raises SIM while WER can lag the no-search baseline.Source: §2 · p. 2
  • Prior depth and width work studies shape, parameterisation, and recurrent iteration, whereas this paper measures refinement and best-of-K search across intelligibility and identity.Source: §2 · p. 3
  • Figure 1 distinguishes training-fixed backbone depth from serve-time refinement steps, with phonemes and a three-second prompt visible as generation begins.The generated region starts fully masked, and the figure presents one training grid as a width–depth–step surface at inference.Source: Figure 1 · p. 3

3 Method

The method trains a masked-diffusion codec TTS grid and evaluates zero-shot synthesis on held-out speakers. It varies refinement at inference while keeping the sampler fixed, and anchors intelligibility and identity measurements to empirical floors.

3 Method

  • The bidirectional transformer predicts Mimi codec tokens using coarse-to-fine masking, with phonemes and a three-second prompt always visible.The backbone has no speaker encoder, making it the only identity pathway; training omits classifier-free guidance.Source: §3 · p. 3
  • Evaluation uses cross-sentence zero-shot synthesis on 400 items from 174 unseen speakers, with real-recording WER and Mimi round-trip similarity as measured floors.The training subset contains 2,000 hours and more than 4,000 speakers; 200 speakers are held out entirely.Source: §3 · p. 3
  • The fixed MaskGIT-style sampler runs T refinement steps per codebook level, yielding 8T function evaluations with shared schedules and keyed generation noise.Source: §3 · p. 3
  • The experiment trains 15 width–depth configurations with three seeds and sweeps T ∈ {1, 2, 4, 8, 16} at evaluation.The grid uses three iso-parameter budgets and five aspect ratios, producing 75 surface points per metric from 45 trained models.Source: §3 · pp. 3–4
  • WER, SIM-o, degeneracy, and UTMOS are evaluated, and weighted nonlinear fits use run-level bootstrap intervals to account for shared models across T settings.Each run's five T points share a trained model, so resampling at the run level preserves their correlation.Source: §3 · p. 4

4 Results

Against measured floors, refinement closes much more of the reachable intelligibility range than the identity range, and this asymmetry persists across training budgets. Identity continues to respond to training compute and selection, while fitted scaling exponents and a single depth–step exchange rate have important scope limits.

4.2 The two capabilities have different bottlenecks

  • 1.84×, 1.36×, and 1.23× are the asymmetry ratios after retraining at the original, 3×, and 6× schedules, respectively, and all remain above one.Across the same budgets, intelligibility’s refinement share saturates while the identity share continues rising; three budgets do not establish whether the ratio ultimately converges above one or to one.Source: §4.2 · p. 4

4.3 The same effect as a fitted exponent, and why we do not lead with it

  • ∆τ = 0.110 (CI [0.092, 0.131]) supports the preregistered exponent contrast in its declared coordinate, but ∆τ changes sign under some non-affine error transformations.The fitted exponent is a local slope over the declared range, and the weighted fit is rejected as a generative model; the paper therefore leads with the model-free reachable-range comparison.Source: §4.3 · p. 5, Table 2 · p. 5

4.1 What refinement buys, measured against floors

  • 86.2% of the reachable intelligibility range but 46.4% of the identity range is closed by refinement from T=1 to T=16, a 1.86× ratio.The comparison is model-free and uses measured references: Whisper’s word error on real recordings and the codec round-trip speaker-similarity ceiling.Source: §4.1 · p. 4, Table 1 · p. 5

4.4 Robustness of the asymmetry

  • The substitution model fits worse than the separable model by ∆AICc = +69.3 for WER and +71.3 for SIM-o, rejecting a single global depth–step exchange rate.The separable fit implies a local κ ≈ 0.41 for WER near the operating point, not a grid-wide design rule.Source: §4.4 · pp. 5–6, Figure 2 · p. 6
  • At fixed parameter count, error depends on model shape, but the anisotropy ratio is unidentified; depth’s effect differs sharply by capability, with βWER = 2.040 versus βSIM = 0.553.The deepest tested shape remains the lower-bound optimum in the two larger budgets.Source: §4.4 · p. 6
  • The latency study does not determine how to divide a fixed T·d budget, and extrapolations from the two smaller budgets do not predict the largest.The frozen degenerate-output rule fires on 0.1% of items at T=16, concentrated at the shallowest shapes.Source: §4.4 · p. 6
  • Training compute raises SIM-o by +0.0827 from 30k to 90k steps and by a further +0.0155 at 180k, remaining the largest tested identity lever.The identity axis is decelerating but still paying, while a codec round trip leaves 0.0747 of headroom above the best system.Source: §4.4 · p. 7
  • At matched NFE, best-of-K search gains +0.0444 SIM-o over refinement at T=64 but incurs +0.0425 WER; refinement has lower WER, 0.1316 versus 0.1955.The search identity advantage is supported across four independent encoder families, though candidates are still selected by one encoder.Source: §4.4 · p. 7, Table 5 · p. 8, Table 6 · p. 8

5 Limitations and outlook

The study’s conclusions are bounded by under-training, codec losses, and a narrow evaluation setup. Some model quantities remain unidentified, and several pre-registered tests failed or were censored by the tested grid.

5 Limitations and outlook

  • 62% of the distance to real speech is lost in the codec before generation, and every model is under-trained, with WER’s saturation step only a lower bound.The grid caps non-embedding parameters at 133 M and training at 30,000 steps; the codec round trip scores 0.555 versus 0.678 for the original waveform.Source: §5 · p. 8
  • Search candidates are selected by one encoder, and compute matching excludes selector and scorer passes, limiting what cross-encoder validation establishes.The other four encoders show the selected candidates are better by their measures, not that they would select the same candidates.Source: §5 · p. 8
  • The study covers one language, codec, and corpus, uses no classifier-free guidance, and sets target length from median corpus character rate rather than per-item prediction.κ remains at its bound in every fit, width amplitude also saturates its bound, and Δρ is therefore undetermined; the Δτ interval widens when fit weights are resampled.Source: §5 · p. 8
  • Four extension and speaker-identity hypotheses were refuted, three were unsupported, two were discordant, and two outcomes remain censored by the grid.The depth optimum reaches the deepest tested shape in the two larger budgets, while WER still falls at the highest fitted refinement step.Source: §5 · p. 8

A Reproducibility

The authors preregistered the protocol and analysis, and released the experiment records and code needed to inspect the results.

A Reproducibility

  • The released artefact includes the harness, all 45 run records, the 75-point surface, and fitting and bootstrap code.Training the grid took 162.5 GPU-hours.Source: A Reproducibility · p. 11
  • The protocol, hypotheses, thresholds, frozen sampler, and analysis plan were registered before the first run.Every gate is logged with its measured values.Source: A Reproducibility · p. 11
  • Training the grid took 162.5 GPU-hours.The released records and code accompany the compute report.Source: A Reproducibility · p. 11

B The shape grid, as run

The appendix reports the shape-grid results and additional inference interventions. Refinement’s effects depend on the metric and schedule, while the fitted exchange between depth and steps remains unsupported.

B The shape grid, as run

  • The pre-registration ledger reports four hypotheses refuted, three unsupported, and two discordant between their pre-specified tests.DISCORDANT is a declared outcome when the primary and secondary tests disagree, not a synonym for refuted.Source: Table 7 · p. 12
  • Speaker-contrastive guidance trades intelligibility for similarity and worsens monotonically with γ; per-item rate matching yields a small mean gain that is unreliable item by item.The table characterizes both as inference-time interventions that do not buy identity.Source: Table 8 · p. 12
  • With 6- and 9-second prompts, outputs remain fluent and speaker-consistent but decouple from the conditioning text; training used 3-second prompts only.The prompt-length sweep uses a fixed 3-second reference for scoring.Source: Table 9 · p. 13
  • 5.2× lower WER results from spending all 32 matched NFE passes on the coarsest rather than finest codebook level, at a cost to identity and naturalness versus a uniform split.The comparison uses the 125 M budget and 15 runs.Source: Table 10 · p. 13
  • Δτ is 0.110 with 95% CI [0.092, 0.131], while κ sits at its bound and the substitution form fits worse than the separable form.The reported comparison means the depth–step exchange rate has no finite estimate; the negative full-versus-N ΔAICc indicates shape matters at fixed N.Source: Table 11 · p. 13
  • Every Δτ interval excludes zero when the fit varies parameter budget, training compute, ASR, or amplitude reparameterisation.The tested changes include a fourth parameter budget and 3× training compute.Source: Table 12 · p. 13
  • 0.952 WER error reduction versus 0.161 SIM-o error reduction over T=1→16 shows metric-selective refinement; WER is still falling at T=16.The figure reports absolute error reductions across the same models, frozen sampler, and items, with only T changed.Source: Figure 4 · p. 13

C Additional figures

The extrapolation figure compares predictions for the largest H-D4 budget under two fitted models. It reports similar prediction errors for SIM-o and WER.

C Additional figures

  • SIM-o extrapolation MAPE is 2.4% versus 2.2%, while WER extrapolation MAPE is 20.2% versus 20.4%, under Mfull and the N-only model, respectively.The two smaller budgets are fitted and the largest budget is predicted.Source: Figure 5 · p. 14

D Refinement-budget allocation across codebook levels

At matched inference compute, codebook-level allocation trades intelligibility against identity and naturalness. The paper also cautions that its raw-scale saturation threshold is not comparable across metrics.

D Refinement-budget allocation across codebook levels

  • The schedules match total NFE = 32 on the 125 M budget, allocating all passes to the coarsest level, the finest level, or evenly.The comparison uses three frozen schedules, 15 runs, and 400 items.Source: D Refinement-budget allocation across… · p. 14
  • Coarse allocation wins WER but loses identity and naturalness against uniform, with intervals excluding zero.The authors characterize this as a trade, not a free win.Source: D Refinement-budget allocation across… · p. 14
  • Saturation is defined as the smallest T whose paired-bootstrap interval for err(T) − err(64) covers zero.This significance-based contrast defines T*.Source: D Refinement-budget allocation across… · p. 14
  • T* = 16 for both metrics under the invariant fraction-of-total-gain definition, while raw-scale T* is not affine-invariant or comparable across metrics.The same 5% band is 0.67% of WER’s range but 11.96% of SIM-o’s; the passage says roughly half the contrast is a precision artefact.Source: D Refinement-budget allocation across… · p. 14

E Fit diagnostics and exploratory analyses

The fit diagnostics expose a binding width-amplitude bound and an exploratory depth-exponent difference. Sensitivity refits reproduce the sign and significance of the step-exponent difference.

E Fit diagnostics and exploratory analyses

  • The width amplitude A reaches its upper bound A ≤ 10 in every fit, so only A w^-α is identified and α is not.The passage says the bound, not the data, is binding; κ reaches its upper bound in every Msub bootstrap replicate, while depth amplitude B is interior.Source: E Fit diagnostics and exploratory… · p. 14
  • βWER = 2.040 versus βSIM = 0.553, a factor of 3.7, is the exploratory depth-exponent separation in the identified parameter.The Part-B ρ result is also exploratory and inherits the width-amplitude cap; the pre-registered test is Part A.Source: E Fit diagnostics and exploratory… · p. 14
  • The points are weighted by 1/SE² using seed-based SE floored at one quarter of pooled seed SD.The floor prevents chance agreement between two seeds from dominating the fit.Source: E Fit diagnostics and exploratory… · p. 15
  • Unweighted refits and a fit including finished runs of a third seed both reproduce the sign and significance of ∆τ.These are exploratory analyses, not part of any declared claim.Source: E Fit diagnostics and exploratory… · p. 15

Use of Generative AI Disclosure

Generative AI assisted with the experimental harness and language refinement, while the authors retained responsibility for the research design, scientific claims, code, and numbers.

Use of Generative AI Disclosure

Related papers

Loading 2610.03320v1…