PaperlayerView exact revision on arXiv

Source-linked AI summary

Depth through recurrence: Looped transformers for flow-matching TTS

Jiabao Ai, Peng Han, Yuchen Song, Zhengjun Yue

arXiv:2609.29768v1eess.AS

TL;DR

Flow-matching TTS repeatedly evaluates a velocity network, raising the question of how to organize recurrent Transformer depth across sampling steps. The paper compares seven weight-sharing layouts with 18 block calls per evaluation. SEQUENCE retains competitive 32-step quality on two datasets with fewer parameters, while reuse placement and sampling budget shape quality trade-offs.

  • Problem

    Flow-matching TTS repeatedly evaluates its velocity network during numerical integration, so the paper asks how to organize recurrence across network depth and sampling steps.

    Source: §1 · p. 1

  • Method

    The study compares seven layouts that vary the amount, order, and placement of weight reuse while keeping 18 block calls per network evaluation.

    Source: §1 · p. 1

  • Results

    On Seed-TTS and LibriSpeech-PC, SEQUENCE retains competitive 32-step quality with 47.1% fewer parameters than Baseline.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Layout selection depends on sampling budget and quality priorities because reuse order and placement produce different quality trade-offs.

    Source: §1 · p. 1

    The evidence covers one model scale and English objective evaluation; future work will add listening tests.

    Source: §5 · p. 4

Abstract

from arXiv · show

We study how to organize Transformer depth through recurrence in flow-matching text-to-speech, varying the amount, order, and place?ment of weight reuse. Seven layouts perform 18 block calls per network evaluation under a common training objective and sam?pler. On Seed-TTS and LibriSpeech-PC, SEQUENCE applies each of nine blocks twice consecutively, retaining competitive intelligibil?ity, speaker similarity, and predicted speech quality at 32 sampling steps with 47.1% fewer parameters than the unshared baseline. Cy?cling six blocks three times further reduces model size but raises 32-step word error rates relative to cycling nine blocks twice. At matched parameter counts and executed depth, reuse order and shar?ing position produce different quality trade-offs. These comparisons depend on sampling budget: Prefix and Suffix have similar 32-step word error rates, but Suffix is worse by 3.44 and 5.97 percentage points at four steps on the two datasets, respectively. Only Middle ranks first or second in mean word error rate at 32 and four steps on both datasets. These results show that the organization of recurrent computation affects synthesis quality, and that reuse layouts should be selected for both the sampling budget and the quality dimensions of interest.

1. INTRODUCTION

The introduction asks how to organize recurrent Transformer depth in flow-matching TTS, where network depth and sampling steps both shape computation. It compares reuse layouts and reports that their quality trade-offs depend on reuse amount, order, position, and sampling budget.

  • Flow-matching TTS repeatedly evaluates its velocity network during numerical integration, making both network depth and sampling steps dimensions of computation.The introduction frames the design question as how to organize block recurrence under repeated network evaluations.Source: §1 · p. 1
  • Prior work studies cross-layer sharing, reuse order, and recurrent computation across language and visual generation; this paper examines their interaction with sampling budget and speech-quality criteria in TTS.Examples include Universal Transformers, ALBERT, Subformer, and visual-generation approaches.Source: §1 · p. 1
  • Seven layouts make 18 block calls per network evaluation while varying sharing amount, reuse order, and the placement of a repeated segment.The comparisons include whole-stack versus adjacent-block reuse and Prefix, Middle, and Suffix placements.Source: §1 · p. 1
  • On Seed-TTS and LibriSpeech-PC, SEQUENCE retains competitive 32-step quality with 47.1% fewer parameters than Baseline.Cycling six blocks three times instead of nine twice worsens 32-step intelligibility; placement also interacts with sampling budget, and only Middle ranks first or second in mean WER across both budgets and datasets.Source: §1 · p. 1

2. RECURRENT DEPTH FOR FLOW-MATCHING TTS

Within each network evaluation, a shared block segment updates the hidden state across visits while weights, flow time, and conditioning remain fixed. The layouts vary how much is shared, the order of reuse, and where the shared segment sits, while holding executed depth at 18 calls.

  • A shared m-block segment Fθ updates the hidden state at each visit within one velocity-network evaluation.Weights, flow time, and conditioning remain fixed across visits, and the hidden state is not reset.Source: §2 · p. 1
  • CYCLE 9 × 2 revisits the whole nine-block stack, while SEQUENCE 9 × 2 reapplies each block immediately; both use 83.6M parameters versus Baseline’s 158.0M.CYCLE 6 × 3 uses six blocks cycled three times and has 58.8M parameters.Source: §2 · p. 1
  • Prefix, Middle, and Suffix place a six-block segment differently around unshared blocks; each has 12 unique blocks, 108.4M parameters, and 18 executed calls.Their respective (p, m, K, q) settings are (0, 6, 2, 6), (3, 6, 2, 3), and (6, 6, 2, 0).Source: §2 · p. 1
  • With executed depth D, sampling steps N, and two CFG branches, the block-call budget is B = 2ND; all seven layouts therefore use B = 36N.Recurrence occurs within each network evaluation, while sampling repeatedly evaluates the network at successive flow times.Source: §2 · pp. 1–2
  • The accompanying layout figure marks repeated IDs as shared weights, uses darker shades for later visits, and shows unshared blocks in gray.Table 1 reports quality scores and inference memory at 500k updates.Source: Figure 1 · p. 2, Table 1 · p. 2

3. EXPERIMENTAL SETUP

The experiments train seven layouts under a shared configuration and evaluate speech quality and resource use at multiple sampling budgets. Uncertainty estimates and call-level diagnostics provide context for interpreting the comparisons.

3.1. Training and controls

  • All layouts use a modified F5-TTS Small implementation and are trained from scratch on LibriTTS with a shared training configuration.The models use width 768; comparisons use 500k-update EMA weights, with one training run per layout.Source: §3.1 · p. 2

3.2. Evaluation

  • Evaluation uses 1,088 Seed-TTS test-en utterances and 1,127 LibriSpeech-PC utterances from 39 speakers, with Euler sampling at 32 and four steps.WER measures mean per-utterance Whisper large-v3 error; SIM-o is prompt/output speaker-embedding cosine similarity, and UTMOS predicts speech quality rather than measuring a listening-test score.Source: §3.2 · pp. 2–3, Figure 2 · p. 3

3.3. Uncertainty and resource measurement

  • Main-table quality means average multiple inference seeds, and paired bootstrap intervals quantify evaluation uncertainty conditional on trained checkpoints, not training-seed variance.Resource measurements use H100, fp32, batch one, and 32-step Seed-TTS synthesis; peak allocated memory includes the vocoder, while sampling RTF excludes vocoding.Source: §3.3 · p. 3

3.4. Diagnostic protocol

  • Diagnostics measure relative hidden-state updates and cosine similarity between visits, then test call sensitivity by replacing a call’s output with its input at every Euler step.Measurements use 128 LibriSpeech-PC utterances across seven flow times and both CFG branches; call removal is tested on Seed-TTS at 16 steps without retraining.Source: §3.4 · p. 3

4. RESULTS AND DISCUSSION

Across the evaluated layouts, SEQUENCE retains competitive objective quality with fewer parameters, while reuse order and placement produce quality trade-offs that vary with sampling budget. Update patterns and call-removal tests further distinguish the layouts.

4.1. Parameter efficiency at matched executed depth

  • SEQUENCE uses 83.6M parameters versus Baseline’s 158.0M (47.1% fewer) at 18 block calls, with competitive objective quality; sampling RTF spans 0.1426–0.1440 across layouts.Peak allocated memory falls from 877 to 574 MB for CYCLE 9 × 2 and SEQUENCE 9 × 2.Source: §4.1 · p. 3
  • CYCLE 6 × 3 saves another 24.8M parameters versus CYCLE 9 × 2 but raises 32-step WER to 2.96/2.65% from 2.60/2.14%.CYCLE 6 × 3 also lowers SIM-o on both datasets, while LibriSpeech-PC UTMOS is 4.06 for both layouts.Source: §4.1 · p. 3

4.2. Reuse order changes the quality trade-off

  • At 32 steps, SEQUENCE lowers Seed-TTS WER from CYCLE’s 2.60 to 2.03 but lowers UTMOS from 3.90 to 3.75; four-step WERs are close.At four steps, CYCLE and SEQUENCE WERs are 9.37 versus 9.32 on Seed-TTS and 11.36 versus 11.46 on LibriSpeech-PC.Source: §4.2 · pp. 3–4

4.3. Placement interacts with the sampling budget

  • At four steps, Suffix exceeds Prefix in WER by 3.44 and 5.97 pp, despite differing by only 0.17 and 0.14 pp at 32 steps.At 32 steps, Suffix is slightly lower; the plotted figure shows WER versus sampling steps for all seven layouts.Source: §4.3 · p. 4, Figure 3 · p. 4
  • Suffix-minus-Prefix WER reverses from +3.52/+6.55 pp at four steps to −0.29/−0.32 pp at eight; only Middle ranks first or second in mean WER across both budgets and datasets.Across layouts, the max–min WER spread shrinks from 4.93/7.74 pp at four steps to 1.29/0.73 pp at 32 steps; Suffix also raises high-error generation rates versus Prefix at four steps.Source: §4.3 · p. 4

4.4. Update structure and call sensitivity

  • Removing Middle’s first two unshared calls raises WER to 28.84% or 99.97%, while removing shared calls 4–14 individually changes WER by at most 1.09 pp.Middle’s first two calls have the largest mean relative updates in every setting.Source: §4.4 · p. 4
  • Mean cosine between two visits’ updates is 0.71 for SEQUENCE, 0.35 for CYCLE, and 0.62 for Middle, the highest among partial loops.SEQUENCE’s visit-update cosines are positive across all 126 block–setting pairs, whereas CYCLE’s block 1 values are negative across all 14 settings.Source: §4.4 · p. 4

5. CONCLUSION

At matched executed depth, looped flow-matching TTS models retain competitive quality with fewer parameters, but layout choice depends on model size, sampling budget, and quality priorities. The evidence is limited to one model scale and English objective evaluation.

  • At matched executed depth, looped flow-matching TTS models retain competitive quality with fewer parameters, while layout choice depends on model size, sampling budget, and quality priorities.Source: §5 · p. 4
  • The evidence covers one model scale and English objective evaluation; future work will add listening tests.Source: §5 · p. 4
Loading 2609.29768v1…