PaperlayerView exact revision on arXiv

Source-linked AI summary

Repetition, Not Length: Isolating the Counting Failure in Neural Text-to-Speech

Kirill Borodin, Vasilii Kudryavtsev, Maxim Maslov, Grach Mkrtchian

arXiv:2609.36974v1cs.CLcs.SD

TL;DR

TTS models can miscount repeated text, and prior reports do not separate repetition from the length it creates. The paper tests matched repeated and control texts across models. Controls are exactly right far more often than repeated inputs at k≥6.

  • Problem

    Prior reports of TTS miscounting repeated text confound repetition with the additional length it creates.

    Source: §1 · p. 1

  • Method

    The study pairs repeated items with same-carrier, same-word-count controls and evaluates them across six checkpoints, two non-autoregressive baselines, and a held-out architecture.

    Source: §2 · p. 1, §1 · p. 1

  • Results

    94.3% of controls versus 18.2% of repeated texts are exactly right at k≥6 across six checkpoints from three architectures.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The measured contraction account does not explain the failure, and why the readout fails remains open.

    Source: §4 · p. 4, §5 · p. 4

    Whether text ordering or restricted vocabulary carries the repetition effect remains unresolved.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Text-to-speech models loop, truncate and lose count on text that repeats a phrase many times. We show that repetition itself is what breaks them, not the length that comes with it. Every repeated sentence in our test set is paired with a control of matched sentence and word count in which no word ever repeats back-to-back. Six models from three architectures render the controls almost perfectly and fail the repeated twins: 94.3% against 18.2% exactly right at k >= 6. The gap survives greedy decoding, repetition-penalty sweeps, four independent speech recognisers and 420 analysis specifications without once reversing sign; a held-out fourth architecture lands within a point of its predicted gap, and one of two non-autoregressive baselines shows the same failure. Varying the period of the text shows the failure grows smoothly with periodicity, half of it surviving when no word is adjacent to itself.

1. INTRODUCTION

TTS models often miscount repeated words, a behavior usually attributed to stability issues and addressed through alignment or decoding changes. This paper separates repetition from length with matched controls and tests the failure across models.

1. INTRODUCTION

  • Repeated words often make TTS models stop early, run on, or loop, a stability problem commonly attributed to exposure bias or missing alignment constraints.Prior fixes target attention supervision or decoding rules.Source: §1 · p. 1
  • Matched controls preserve the repeated text’s carrier and word count while replacing repeated copies with distinct fillers, separating repetition from length.If length alone explains the difficulty, the repeated and control curves should coincide.Source: §1 · p. 1
  • The study combines a controlled manipulation across six checkpoints, two non-autoregressive baselines, and a held-out architecture with a purpose-built test set and speech-judging method.It also reports that the contraction account of [8] does not describe the measured decoders.Source: §1 · p. 1

2. EXPERIMENTAL SETUP

The setup pairs authored repeated-word items with length-matched controls, evaluates a multi-architecture model panel using a speech recogniser, and reports count-based metrics with uncertainty checks.

2. EXPERIMENTAL SETUP

  • The 180-item authored test set pairs each repeated item at k≥2 with a same-template, same-word-count control using distinct fillers.Targets span k values from 1 to 32; filler pools cycle above k=8, and a never-cycled variant quantifies that control design difference.Source: §2 · p. 1
  • The panel contains six Llasa, Qwen3-TTS, and XTTS-v2 checkpoints, with VITS and F5-TTS as non-autoregressive baselines and CosyVoice 2 held out as a fourth architecture.Each system renders every item at three seeds using its shipped decoding defaults.Source: §2 · p. 1
  • A greedy CTC wav2vec2 recogniser transcribes the audio for counting; on the audited concatenated utterances, it recovers the full count while Whisper large-v3 recovers a median 0.25 on trials where it returns anything.Whisper returns nothing on the other 60 of 108 trials; its autoregressive decoder and language-model prior are unsuitable for this audio.Source: §2 · p. 2
  • The metrics are relative count error, exact-count rate, and the control-minus-repeated exact-rate gap, with exclusions blind to count and each analysis reporting its own n.The headline analyses cover the word-repetition ladder and its controls.Source: §2 · p. 2
  • Headline estimates are computed four ways and tested under 420 specifications, with 95% confidence intervals and checkpoint-level bootstrap for panel summaries.The checkpoint is the unit of generalization because generations cluster within checkpoints.Source: §2 · p. 2

3. RESULTS

Length-matched controls stay near-perfect while repeated text loses exact counts, and the deficit grows smoothly with periodicity. The gap persists across analysis and decoding variations, appears in a held-out architecture and one non-autoregressive baseline, and accompanies slower growth of distinguishable decoder states.

3.1. The limit tracks repetition, not length

  • 94.3% of controls versus 18.2% of repeated items are exactly right at k≥6; controls stay near-perfect while repeated-item accuracy collapses.The curves do not separate below k=6.Source: §3.1 · p. 2, Figure 1 · p. 2, Table 2 · p. 3
  • Across 420 analysis specifications, the gap spans 34.4–86.4 points and never reverses sign; CosyVoice 2 lands within a point of its predicted 64.4-point gap.Using never-cycled fillers throughout still yields a 73.1-point gap in the matched-count re-generation.Source: §3.1 · p. 2

3.2. The deficit is graded in the period

  • The deficit is graded with period: exact rates rise from 10.4% at p=1 to 93.1% at p=8, with half the deficit remaining at p=2.At p=2, no token is adjacent to itself; adding commas, full stops, or “and” recovers none of the deficit.Source: §3.2 · p. 3, Table 5 · p. 3
  • The periodicity ladder varies period while holding carrier, word count, and requested count fixed, but whether word ordering or restricted vocabulary carries the effect remains unresolved.Shuffling each item’s words recovers almost none of the pooled gap, despite divergent checkpoint outcomes.Source: §3.2 · p. 3

3.3. Not the decoding rule or our analysis choices

  • Greedy decoding, repetition-aware sampling, penalty sweeps, and four independent recognisers leave the gap positive; with every exclusion rule off, it is 61.7 points.The token-budget exclusion is conservative because budget hits can only cut counts short.Source: §3.3 · p. 3, Table 6 · p. 4

3.4. Non-autoregressive baselines

  • F5-TTS has a 60.0-point repeated–control gap, while VITS shows no dissociation; oracle total duration helps F5-TTS below k=12 but not at or above it.Even with the right total duration, F5-TTS cannot place thirty-two repetitions inside it.Source: §3.4 · p. 3, Table 7 · p. 4

3.5. Repetition slows the growth of the state

  • Repeated text grows distinguishable decoder states at 0.46 of the control rate, with positive capacity gain in five of six checkpoints, indicating slowing rather than a halt.The checkpoint-level gain gap is 24.7 [13.7, 36.5] states per unit log k.Source: §3.5 · pp. 3–4

4. DISCUSSION

The measured decoder dynamics do not support the contraction account: states expand, and attention findings run opposite to its prediction. Count information is often decodable even when rendering fails, but the state-versus-readout question remains unresolved.

  • Across five checkpoints, q ranges from 21.0 to 347.7, indicating expansive dynamics on repeated and control text rather than contraction toward a fixed point.The checkpoint with zero median count error has q = 24.7.Source: §4 · p. 4
  • Flatter attention predicts better counting (r = 0.59), opposite to the contraction account’s prediction that attention would become near-uniform over repeated copies.The logit spread is 1.23 nats, a 3.4-fold ratio.Source: §4 · p. 4
  • Late hidden states retain count information while rendering fails (median R2 = 0.48), but beyond k = 48–128 that signal weakens or may reflect audio length.One checkpoint does not beat a constant predictor, another ties it at 0.99, and the proposed rank-1 patches fail their positive control, leaving the state-versus-readout question open.Source: §4 · p. 4, Table 9 · p. 4
  • Under a length-matched control, representational collapse slows rather than reaching a fixed point, while the effect varies with input period unlike model-invariant counting limits.Source: §4 · p. 4
  • The study does not separate repetition from training-set frequency, validates its judge on audio rather than listeners, and tests a compute-limited panel of six checkpoints in three families.Source: §4 · p. 4

5. CONCLUSION

With input length held fixed, repetition alone degrades counting across tested autoregressive systems. The gap also appears in a held-out architecture and one parallel decoder, while its mechanism remains unresolved.

  • With length held fixed, repetition alone reduces exact rendering in every tested autoregressive text-to-speech system: controls are exactly right in 94.3%, repeated text in 18.2%, a 76.1-point gap across six checkpoints from three architectures.The per-checkpoint gap is 76.7 points.Source: §5 · p. 4
  • 34.4–86.4 points across 420 analysis specifications never reverse the gap, and half of it remains when adjacent repeats are removed.The held-out architecture shows a 64.4-point gap, as predicted; one of two parallel decoders also fails, so the effect is not limited to autoregression.Source: §5 · p. 4
  • Repeated text grows distinguishable states at 0.46 of the control’s rate, while the proposed attractor mechanism fails its own measurement with q never below 1.Why the readout fails remains open.Source: §5 · p. 4

COMPLIANCE WITH ETHICAL STANDARDS

The study used released text-to-speech checkpoints and synthetic author-written text, without human or animal subjects or personal data.

Supplementary Material

The supplement details the test set, matched controls, instrumented subset, and scoring rules used to evaluate repetitive speech.

A. THE TEST SET IN FULL

  • The test set has 180 items in five families, with word repetition and controls tested across a k ladder from 1 to 32.Sentence repetition stops at k = 16, while tongue twisters stop at k = 4; number phrases use fixed strings rather than a ladder.Source: A. THE TEST SET IN FULL · p. 6
  • Number phrases test place-value counting, while sentence and tongue-twister items count specified target words, with two twister targets probing stress outside the counting metric.Word-repetition targets are common, short, phonetically distinct from fillers, and homophone-free.Source: A. THE TEST SET IN FULL · p. 6
  • Matched controls preserve prefix, suffix, and word count while replacing target copies with distinct fillers; hidden-state and attention capture covers 136 word- and sentence-repetition and control items.Boundary lists are explicit and model-agnostic, and twisters and number phrases are behavioural-only.Source: A. THE TEST SET IN FULL · p. 6

B. SCORING

  • The scoring setup replaces Whisper’s undercounting autoregressive judge with a CTC judge and defines exact transcript counts, duration-calibrated counts, and an exact-correct criterion.Count B is calibrated on trusted low-k items; an item is correct only when Count A matches the expected count and the duration ratio is 0.70–1.45.Source: B. SCORING · p. 6
  • Empty or degenerate audio is reported separately, and large disagreements between transcript and duration counts are flagged rather than converted into a synthesized number.Degenerate audio includes duration below 0.25 seconds, RMS below 10^-3, spectral flatness above 0.35, or an empty transcript.Source: B. SCORING · p. 6

C. THE JUDGE IS PART OF THE EXPERIMENT

Whisper undercounts or omits repeated audio, so the study replaces it with a CTC judge and tests the same audio with multiple independent recognisers. The results support a repetition-related gap, while checkpoint-level inference emphasizes effect size and consistency over multiplicity-corrected significance.

C. THE JUDGE IS PART OF THE EXPERIMENT

  • On identical synthetic audio, the CTC recogniser scores a 1.00 median counted/true ratio, while Whisper scores 0.25 on returned transcripts and produces none for 𝑘=16 or 32.Whisper’s autoregressive language-model prior discounts verbatim repeats, making it unsuitable as the counting judge for this failure mode.Source: C. THE JUDGE IS PART OF THE EXPERIMENT · p. 6, Table S5 · p. 7

C.1. Four independent judges on the full population

  • All four judges show a positive gap in six of six checkpoints and three of three families, with the weakest retaining 85% of the published gap.The repeated-arm exact rate varies by only 1.0 point across judges, while the control rate varies by 12.2; the CTC-specific collapse confound can inflate but not manufacture the gap.Source: §C.1 · p. 8
  • The Spanish dissociation is untested because its control exact rate was 20.4%, below the pre-registered 50% rendering-quality floor.On constructed Spanish ground truth, CTC scored 1.00 and Whisper 0.19 where it returned a transcript, but Whisper returned nothing at the two highest 𝑘 values.Source: §C.1 · p. 8

D. CHECKPOINT-LEVEL INFERENCE AND THE SPECIFICATION CURVE

  • The exact-rate gap is positive in all six checkpoints and never below 60 points, but the small panel limits multiplicity-corrected significance.At 𝑛=6 the smallest two-sided exact 𝑝-value is 0.031, and Holm’s floor is 0.094; the authors therefore emphasize effect sizes and consistency.Source: D. CHECKPOINT-LEVEL INFERENCE AND THE… · p. 8

D.1. The clustering, modelled instead of counted

The authors model checkpoint clustering rather than treating generations as independent, and compare several interval and Bayesian approaches. The observed gap remains positive across the reported specifications, while prediction for a new family remains uncertain.

D.1. The clustering, modelled instead of counted

  • Across clustering-respecting models, the arm effect stays near 76 points and every interval excludes zero.The six checkpoints belong to three families, so treating them as independent replicates would double-count shared structure.Source: §D.1 · p. 8
  • The checkpoint-clustered OLS interval is the widest of six reported intervals, all of which exclude zero; the Wald interval is excluded as an artifact of weak variance identification.Three families barely identify the family-varying arm-slope variance, so shrinkage understates between-family spread.Source: §D.1 · p. 8
  • The posterior predictive gap for a new family is wide enough that it does not exclude a deficit-free fourth architecture.With random effects set to zero, the panel gap is 74.9 points [41.5, 89.4], below intervals that condition away between-family variance.Source: §D.1 · p. 8
  • The widest prior allows a one-in-twenty-five chance that a fourth family has no deficit or reverses it, reflecting variance estimated from only three families.Every prior still leaves a large effect in every observed family.Source: §D.1 · p. 8
  • Across 420 analysis specifications, the exact-rate spread is 2.5× without a sign change, and the median-error gap is never negative.The two outcomes use different scales and should not be averaged.Source: §D.1 · p. 8

D.2. What the brackets mean

  • Reported intervals are 95% ranges chosen by estimand; item-pooled intervals are too narrow because they ignore within-checkpoint clustering.Panel summaries use a nonparametric bootstrap over checkpoints, the unit the claims generalize over.Source: §D.2 · p. 8

E. THE SHAPE OF THE DEFICIT

The deficit’s shape favors proportional rather than saturating rendered counts within the tested ladder, but the horizon analysis and exclusions have important qualifications. A weak checkpoint is the exception, and reported exclusions shrink rather than create the gap.

E. THE SHAPE OF THE DEFICIT

  • Proportional growth wins in five of six checkpoints and pooled, by orders of magnitude in residual, over the saturating alternative.The soft-horizon fit places its horizon at or beyond the ladder’s top in all six checkpoints, which does not test a horizon beyond the observed range.Source: E. THE SHAPE OF THE DEFICIT · pp. 8, 10

E.1. Fitted-form dependence of the horizon ratio

  • The horizon-ratio direction is consistent across three saturating curve families, but its magnitude depends on the fitted form.After the Qwen budget repair, the soft-horizon ratio moves from 3.5 to 4.2 and the three-family range from 2.7–4.3 to 3.1–5.4; all three forms saturate by construction.Source: §E.1 · p. 10, Table S16 · p. 10

E.2. Do our own exclusions make the gap?

  • Exclusions flatter the gap by about fourteen points rather than creating it; with no exclusions, the gap remains positive.The template rule shrinks the gap by restoring control items that the judge cannot transcribe, while the potentially gap-flattering budget rule does not explain it.Source: §E.2 · pp. 10–11, Table S12 · p. 11, Table S11 · p. 11

E.3. The inverting checkpoint is the panel’s weakest

  • Qwen3-TTS-1.7B is the only checkpoint without repeated-side saturation through 𝑘=128 and with an inverted horizon ratio, and it is also the panel’s weakest checkpoint.Its generation budget prevents testing whether the horizon lies beyond 𝑘=128, so the longer-ladder possibility remains unresolved.Source: §E.3 · p. 11

F. ALTERNATIVE EXPLANATIONS, ANSWERED WITH MEASUREMENTS

The paper measures several alternative explanations and decoding changes against the repetition-related gap. The tested controls and alternatives do not overturn the reported pattern, though several checks are limited in scope or power.

F. ALTERNATIVE EXPLANATIONS, ANSWERED WITH MEASUREMENTS

G.1. The repetition penalty

G.2. Greedy decoding

  • Greedy decoding leaves the Qwen3-TTS-0.6B gap within three points of sampled decoding, with no cap-hit generations in either arm.This check covers one checkpoint and one alternative decoding mode; beam search remains untested.Source: §G.2 · p. 14, Table S24 · p. 14

G.3. Repetition-aware sampling

  • Repetition-aware sampling leaves the deficit above the pre-committed survives band, while the control arm remains 100.0% exact.The test uses one checkpoint and a reimplementation from the published description rather than the authors’ code.Source: §G.3 · p. 14, Table S25 · p. 14

H. THE PERIOD LADDER IN FULL

At fixed carrier, word count, and requested count, exact rates rise monotonically as repetition period increases, with half the deficit remaining at period 2. Shuffling and boundary-marking tests qualify the periodicity account, which varies across architectures.

The full ladder

  • Exact renderings rise monotonically with period in every checkpoint, and half the deficit survives at period 2, where no token is adjacent to itself; this ladder varies only the period.Source: H. THE PERIOD LADDER IN FULL · p. 14

The disambiguation arm

  • Boundary punctuation restores only 0.9% of the deficit, and the length-matched recovery fraction Rmatched = 0.009 falls below the pre-committed 0.15 falsification threshold for individuation.The comparison holds period 1 while changing only the boundary; the and-conjoined variant has R = 0.047 and is not averaged into Rmatched.Source: H. THE PERIOD LADDER IN FULL · p. 15

Odd periods

  • Both pre-committed odd-period tests pass, with ordered-scan rates monotone across periods 1, 2, 3, 4, 6, and 8 under four counting rules.The upper inequality in Test B is not separable from noise because the period-6-to-8 gap includes the saturated ladder top.Source: H. THE PERIOD LADDER IN FULL · p. 15

Shuffled order at fixed composition

  • Shuffling supports periodic ordering on two checkpoints, but not all: it recovers most of the deficit on both Qwen checkpoints, none on Llasa-1B, and destroys XTTS-v2.The test was designed to distinguish periodic ordering from a low type-token-ratio account.Source: H. THE PERIOD LADDER IN FULL · p. 15
  • The period-2 shuffle is uninformative because any non-periodic ordering of the two token types must contain adjacent identical tokens; period 8 has no deficit to recover.The period-4 shuffle is decisive because zero adjacency is constructible and was achieved exactly.Source: H. THE PERIOD LADDER IN FULL · p. 15
  • At period 4, shuffling fixed-composition fillers recovers nothing pooled, but dropping XTTS-v2 yields R(4) = +0.473, revealing an architecture split.Three of four checkpoint intervals exclude zero in opposite directions; the authors decline the placebo correction because its pre-committed threshold was not met.Source: H. THE PERIOD LADDER IN FULL · p. 15

Architecture scope

  • The title’s periodicity account holds on two checkpoints, fails on a third, and reverses on a fourth, while the length-matched dissociation holds across all six panel checkpoints and a held-out architecture.The authors therefore treat periodicity as architecture-dependent and retain repetition, not length, as the claim that survives across systems.Source: H. THE PERIOD LADDER IN FULL · p. 15

I. THE NON-AUTOREGRESSIVE BASELINES

The non-autoregressive baselines disagree: F5-TTS shows a repetition deficit, while VITS does not. The contraction account is rejected by the measured decoder maps, but the later causal intervention cannot support a causal conclusion.

I. THE NON-AUTOREGRESSIVE BASELINES

  • One non-autoregressive baseline reproduces the deficit and one does not: F5-TTS shows a panel-like gap, while VITS shows no dissociation and repeated items are ordered oppositely.F5-TTS durations scale with k and its controls remain above 83%; VITS’s null is not a floor effect or a stock-configuration artifact.Source: §I · p. 16, Table S33 · p. 16, Table S32 · p. 17
  • Supplying F5-TTS the correct total duration raises exact rate from 43.3% to 76.7% below k = 12, but yields 0.0-point gain at or above k = 12.At the severe end, the model still cannot place thirty-two repetitions inside the supplied duration; the experiment uses one model and 15 items per cell.Source: §I · pp. 16–17, Table S35 · p. 18

J. STATE INSTRUMENTATION

  • Hidden-state probes predict log2 k with selected-layer leave-one-template-out R2 values from 0.18 to 0.71 across 24 checkpoint–family–window cells.The probe uses mean-pooled states, ridge regression, and a fixed penalty; generation instrumentation was validated against normal generation before interpretation.Source: J. STATE INSTRUMENTATION · p. 17

K. THE CONTRACTION PREMISE, MEASURED

L. DILUTION AT THE ITEM LEVEL, AND PROBING PAST THE LADDER

  • The item-level attention prediction reverses: within model-count cells, flatter-attention items are counted better, while controls show no correlation.Past the ladder, three Llasa probes beat the length-only baseline, but both Qwen recoveries are voided because trajectory length alone predicts count better and budget truncation is commoner on repeated items.Source: L. DILUTION AT THE ITEM LEVEL, AND… · pp. 18, 20, Table S38 · p. 19, Table S39 · p. 19

M. CAUSAL INTERVENTIONS ON THE COUNT

Loading 2609.36974v1…