PaperlayerView exact revision on arXiv

Source-linked AI summary

TTS-Q: Perception-Guided Post-Training Quantization of Text-to-Speech Diffusion Transformers

Xiaoqin Feng, Yuqi Li, Weilun Feng, Yi-Cheng Lin, Kai Li, Xianglong Wang, Kuo Yang, Chuanguang Yang, Yingli Tian, Wei Li, Zhongwei Cheng, Lin Chen

arXiv:2610.12474v1cs.SDcs.LGeess.AS

TL;DR

Zero-shot TTS diffusion transformers can clone a voice but their size and multi-step sampling challenge constrained-device use, while 4-bit PTQ can preserve intelligibility yet erode speaker timbre. TTS-Q uses hard-item task-matched calibration and budget-balanced higher precision for late-step FFN activations; across 1B and 3.5B models, it recovers speaker-similarity deficit without material CER/WER changes.

  • Problem

    For TTS diffusion transformers, prior PTQ methods had not been studied, and 4-bit quantization could keep intelligibility near full precision while eroding speaker timbre.

    Source: §1 · p. 1

  • Method

    TTS-Q selects hard-to-quantize items from task-matched calibration candidates and allocates higher activation precision to late-step FFNs under a balanced nominal budget.

    Source: §1 · p. 1, Structure-Aligned Precision Allocation · p. 4

  • Results

    TTS-Q recovers 34% and 80% of the summed speaker-similarity deficit at 1B and 3.5B, respectively, with every CER/WER change within 0.105 percentage points of the lossmax-calibrated uniform-W4A4 reference.

    Source: §5 · p. 7

  • Takeaways & Limitations

    Low-bit TTS should be designed and evaluated around both linguistic content and speaker identity, rather than treating intelligibility alone as sufficient.

    Source: §5 · p. 7

    Reported differences are small point estimates without confidence intervals or listening-test validation, and calibration under an unknown target-task distribution remains untested.

    Source: §5 · p. 7

Abstract

from arXiv · show

Zero-shot text-to-speech (TTS) can clone a voice from seconds of audio. Yet the quality comes from billion-parameter diffusion transformers whose size and multi-step sampling put them out of reach of the on-device, assistive, and real-time applications that need them most. Post-training quantization (PTQ) shrinks such models without retraining, but it has been built for language and vision, not speech. At 4 bits, a strong quantizer leaves intelligibility close to full precision but consistently erodes speaker timbre, so error rates stay low while the voice drifts from the target speaker. We ask why quantization hurts speech, where the damage lives, and whether it can be repaired without retraining. To find out, we run the first 4-bit weight-and-activation (W4A4) PTQ study of a TTS diffusion transformer at 1B and 3.5B. The timbre cost comes from both the weight and the activation quantization, but only the activation part has the timestep structure a precision schedule can exploit. We therefore introduce TTS-Q, which selects hard-to-quantize items from task-matched calibration candidates and applies a budget-balanced mixed-precision schedule that assigns higher precision to late-step feed-forward activations. TTS-Q recovers timbre best at both scales, at no cost to intelligibility. Separately, the deployed calibrated-W4A4 reference reduces peak memory on GPU, providing a practical reference for resource-constrained TTS deployment.

1 Introduction

Zero-shot TTS diffusion transformers offer high-quality voice cloning but are too large and slow for constrained devices. This study examines why 4-bit quantization harms speaker timbre and introduces TTS-Q to address that damage without retraining.

Motivation

  • Billion-parameter diffusion transformers require multi-step sampling, making them too large and slow for assistive, offline, and real-time applications on constrained devices.These applications need models small and fast enough for on-device use.Source: §1 · p. 1
  • At W4A4, a strong quantizer keeps intelligibility close to full precision but consistently erodes the speaker timbre that makes a cloned voice sound like its target.The study asks where the damage occurs and whether it can be repaired without retraining.Source: §1 · p. 1

Diagnosis

  • Across 1B and 3.5B models, the timbre damage is concentrated in late-step feed-forward layers, while emphasizing poorly fitted blocks or speaker similarity during calibration does not restore it.The study uses FlatQuant as its fixed PTQ optimizer and strictly paired audio comparisons.Source: §1 · p. 1

Contributions

  • TTS-Q combines task-matched hard-item calibration with a budget-balanced schedule that raises late-step FFN activation precision, restoring speaker timbre at both model sizes without reducing intelligibility.The deployed FlatQuant-W4A4 reference separately reduces peak GPU memory by 2.1×–3.6×.Source: §1 · pp. 1–2

2 Related Work

Zero-shot TTS uses autoregressive or diffusion/flow-matching backbones, with this work focused on multi-step diffusion transformers. Prior quantization research covers language, image, video, and other audio models, but the supplied related work identifies a gap for diffusion TTS.

Zero-shot TTS and diffusion backbones

  • Zero-shot TTS systems either predict discrete speech tokens autoregressively or refine latent acoustic representations over multiple diffusion or flow-matching sampling steps.The paper studies LongCat-AudioDiT, a billion-parameter diffusion transformer using a multi-step ODE solver.Source: Zero-shot TTS and diffusion backbones · p. 2

Post-training quantization

  • Language-model PTQ addresses activation outliers with rescaling or transforms, while diffusion PTQ adapts quantization across changing sampler steps and layers; none had been tested on a TTS diffusion transformer.The paper uses FlatQuant as its reconstruction-based PTQ optimizer.Source: Post-training quantization · p. 2
  • Prior speech quantization covers neural vocoders, quantization-aware retraining, general audio diffusion, and autoregressive voice cloning, but not a diffusion TTS backbone or joint intelligibility and speaker-similarity measurement.The paper presents its work as the first study of that joint evaluation in this setting.Source: Post-training quantization · p. 2

3 Method

TTS-Q combines offline calibration-set construction and quantizer fitting with inference-time activation precision allocation. It selects difficult task-matched items and concentrates higher precision on late-step FFNs while retaining W4 weights.

3 Method

  • TTS-Q’s two stages build task-matched calibration candidates and select hard items offline, then apply budget-balanced mixed precision to late-step feed-forward activations at inference.Selected item-step pairs form snapshots for uniform-W4A4 fitting before inference-time allocation.Source: §3 · p. 2, Figure 1 · p. 3

Preliminaries

  • The TTS DiT is conditioned on target text and a reference utterance, iteratively samples a latent over S steps, and uses a VAE to decode the final latent into audio.Each block contains self-attention, text cross-attention, and an FFN; adaptive projected guidance combines conditional and unconditional evaluations.Source: Preliminaries · p. 2
  • The reconstruction-based blockwise PTQ optimizer quantizes every linear projection in self-attention, cross-attention, and FFNs; WwAa denotes weight and activation bit-widths.The target uniform configuration is W4A4.Source: Preliminaries · p. 2

Task-Mirrored Calibration and Loss-Guided Selection

  • Selected calibration items contribute the same sampler coordinates to reconstruction snapshots, whose item-step pairs form the data used to fit the quantizer.The snapshot count is Ncal times the number of retained sampler steps.Source: Task-Mirrored Calibration and… · pp. 3–4, Preliminaries · p. 3
  • Task-mirrored candidate sets use aggregate task descriptors and cross-paired prompts and targets, while loss-guided selection retains items with the largest initialization-time reconstruction errors.Candidate construction excludes evaluation-set text, audio, speakers, and model outputs; item ranking is scale-specific.Source: Task-Mirrored Calibration and… · pp. 3–4

Structure-Aligned Precision Allocation

  • At inference, TTS-Q keeps weights at W4 and reallocates activation precision toward late-step FFNs under a parameter-weighted nominal budget matched as closely as possible to uniform A4.The allocation is fixed during inference, and nominal-budget matching does not imply equal measured latency.Source: Structure-Aligned Precision Allocation · p. 4

4 Experiments

The experiments evaluate W4A4 quality and deployment efficiency across two model scales, then test calibration selection and activation-allocation strategies. TTS-Q improves speaker-similarity estimates with little change in text-error rates, while the deployed W4A4 reference reduces peak memory.

Experimental Setup

  • The study uses FlatQuant on 1B and 3.5B LongCat-AudioDiT models, paired fixed-seed evaluation subsets, and CER/WER and SIM metrics.The TTS-Q allocation keeps weights at W4 and applies A9 to the final five FFN steps and A3 elsewhere; evaluated calibrated methods use task-specific optimization settings.Source: Experimental Setup · pp. 4–5

Main Results

  • Lossmax gives FlatQuant the strongest overall profile at both scales, while task-best or uniform-random remains better on some subset-level metrics.Direct rounding and A4 adaptations of weight-only or W8A8 methods incur larger text-error increases than Q-DiT, QuaRot, and FlatQuant variants.Source: Main Results · pp. 5–6
  • TTS-Q raises unweighted summed SIM by 0.0046/0.0041 at 1B/3.5B, with every CER/WER change within 0.105 percentage points of lossmax W4A4.The held-out comparison keeps the calibration items and W4 weights fixed; the reported sums are unweighted across the three evaluation subsets.Source: Table 1 · p. 5, Main Results · p. 6

Calibration-Data Selection

  • Lossmax selection yields the highest development text score, and late2 snapshots produce higher summed SIM than first+last at both scales.The snapshot study holds its offline budget at 64 snapshots; the late2 choice uses 32 lossmax items at steps {13, 14}.Source: Calibration-Data Selection · p. 6

Structure-Aligned Activation Allocation

  • Late restoration outperforms early restoration, and FFN restoration outperforms self-attention restoration in all six subset-by-scale comparisons.In the matched-budget temporal comparison, Late-FFN-balanced has higher SIM and lower CER/WER than Early-FFN-balanced in all six comparisons.Source: Structure-Aligned Activation Allocation · p. 6, Figure 3 · p. 7

Deployment Efficiency

  • The deployed W4A4 reference reduces peak allocated memory by 2.1×/3.6× at 1B/3.5B and gives a 1.439× speedup on the 3.5B long input.The deployment measurements compare the reference path with matched FP32 on one A100; W4A4 is slower at the other reported input conditions.Source: Deployment Efficiency · p. 6, Figure 4 · p. 7

5 Conclusion and Limitations

TTS-Q combines loss-guided calibration with a budget-balanced late-FFN activation schedule, recovering speaker-similarity deficit at both model scales while keeping text-error changes small. The evaluation remains limited by point estimates without confidence intervals or listening validation, alongside untested deployment and generalization settings.

Conclusion

  • The observed quantization gap involves both weights and activations, while the tested structure-aligned recovery acts on activation precision.The conclusion recommends evaluating low-bit TTS on both linguistic content and speaker identity, rather than intelligibility alone.Source: §5 · p. 7
  • The late-FFN allocation recovers 34%/80% of the summed SIM deficit at 1B/3.5B, with every CER/WER change within 0.105 percentage points of the uniform-W4A4 reference.The pipeline forms the calibration set through task-mirrored candidates and loss-guided selection, then applies the activation schedule at inference.Source: §5 · p. 7

Limitations

  • Reported differences are small point estimates without confidence intervals or perceptual validation from listening tests.Other scope limits include one TTS DiT family, two model scales, three language subsets, and no benchmark of the mixed-activation TTS-Q deployment path.Source: §5 · p. 7

A Implementation and Activation-Allocation Details

The implementation uses calibrated FlatQuant-W4A4 as its uniform reference and reuses its learned quantization parameters for the mixed-precision quality evaluation. TTS-Q raises precision only for late-step FFN activations, while calibration snapshots, sampler settings, and the SIM evaluator are specified separately.

Implementation

  • The reference quantizes LongCat-AudioDiT-1B and -3.5B with FlatQuant; its uniform-W4A4 calibration uses lossmax items and late2 snapshots.The models have 24 and 32 blocks, respectively, and FlatQuant uses 200 reconstruction iterations per block.Source: A Implementation and… · p. 10

Activation Allocation

  • TTS-Q keeps all weights at W4, assigning A9 to FFN activations in the final five sampler steps and A3 to remaining quantized activation sites.The A4-to-A32 restoration arms are diagnostics only; no A32 site is used in the final schedule.Source: A Implementation and… · p. 10
  • Mixed-precision evaluation reuses the uniform-W4A4 model’s learned transforms and clipping parameters without bit-specific refitting.The same dynamic asymmetric per-token activation quantizer is used at A3, A4, and A9; bit width changes the number of quantization levels.Source: A Implementation and… · p. 10

Implementation

  • The released sampler uses 16 ODE nodes and adaptive projected guidance; item-ranking steps, calibration snapshots, and inference windows serve distinct roles.The sampler has 15 explicit-Euler steps; indices 0–14 score items, fit the quantizer, or assign inference precision, respectively.Source: A Implementation and… · p. 10

Evaluation Implementation

  • SIM is cosine similarity between synthesized and reference-prompt embeddings from a speaker verifier combining WavLM-large states with an ECAPA-TDNN head.The verifier produces a final 256-dimensional embedding rather than selecting a single WavLM layer.Source: A Implementation and… · p. 10

A.1 Activation-Allocation Arm Definitions

The allocation diagnostics compare where and when higher activation precision is restored, then validate a budget-balanced early-versus-late FFN schedule. The experiments use restoration controls and a parameter-weighted nominal activation budget, not a measured hardware-cost model.

A.1 Activation-Allocation Arm Definitions

  • Restoration diagnostics keep weights at W4 and change selected activation inputs from A4 to A32, targeting early or late steps or all-step self-attention or FFN activations.Late-FFN targets the intersection of FFN activations and steps 10–14; a budget-matched late-nonFFN mirror is absent.Source: §A.1 · pp. 10–11
  • Development pilots selected late over early and FFN over self-attention by summed-SIM direction at both scales, then froze late-FFN placement before held-out evaluation.The directional rule rejected choices with summed text-accuracy regression at both scales; it did not assert joint Pareto dominance.Source: §A.1 · pp. 10–11
  • The matched-budget temporal controls assign A9 to FFN activations in either steps 0–4 or 10–14 and A3 elsewhere; both match uniform A4 nominally at 1B and use 3.947 bits versus 4.000 at 3.5B.The controls differ only in the five-step window; these nominal values are not measured latency or memory budgets.Source: §A.1 · p. 11
  • The nominal budget is parameter-weighted over a K-of-S-step module window, and the chosen (A9, A3) pair yields 4.000 bits at f=1/2 and 3.947 bits at f=9/19.At f=1/2, exact matching to uniform A4 constrains the integer pair to ahi + 5alo = 24; (9, 3) is chosen over alternatives that put most activation sites at two or one bit.Source: §A.1 · p. 11
  • Equation S1 is parameter-weighted accounting, not a hardware cost model; A3 and A9 are not native tensor-core formats, so mixed-precision results use fake quantization.The deployment study benchmarks only the deployed FlatQuant-W4A4 reference path.Source: §A.1 · p. 11

B External Calibration Pool and Selection Protocol

The calibration pool uses external, task-informed data and disjoint development and held-out evaluation splits. Candidate sets are frozen for comparison, while baseline W4A4 adaptations are evaluated under a common nominal interface.

B External Calibration Pool and Selection Protocol

  • The external pool contains 376 items from 158 speakers, sourced from AISHELL-3 and LibriTTS, with hard-style Mandarin targets built deterministically from authentic transcripts and toneless Pinyin.Its construction uses source-corpus material and approximates relevant surface-form difficulty without reusing evaluation text.Source: B External Calibration Pool and… · p. 11
  • Pool construction uses benchmark descriptor ranges to guide task-form coverage, not exact distribution matching; the realized English prompt duration is 5.16 [5.03, 5.33] s versus the benchmark’s 4.53 [3.49, 6.08] s.Regular Mandarin target length and maximum syllable repetition are closely aligned, and hard-style targets preserve longer targets and elevated repetition.Source: B External Calibration Pool and… · pp. 11–12
  • Evaluation items are excluded from calibration and model selection, with corpus-provenance separation and five automated overlap and manifest checks required before pool use.Prompts and targets are cross-paired so no target text comes from the prompt’s own utterance or speaker.Source: B External Calibration Pool and… · p. 12
  • Three mutually disjoint 32-item task-matched candidate sets form a 96-item pool for loss-guided selection, and the task-best set is designated separately at each scale.Lossmax and lossmin are selected from the pooled candidates by initialization reconstruction loss; their memberships can differ by scale, while the uniform-random reference is drawn from all 376 pool items.Source: B External Calibration Pool and… · p. 12

C Baseline W4A4 Adaptations

  • The study evaluates methods on the same held-out remainders under a common nominal W4A4 interface; GPTQ, SmoothQuant, and AWQ use compatibility adaptations rather than native activation settings.Calibrated configurations use scale-specific lossmax items and late2 snapshots, except explicitly named uniform-random and task-best FlatQuant ablations.Source: C Baseline W4A4 Adaptations · p. 12

D Additional Calibration and Snapshot Results

Calibration-set selection improves the development text score without eliminating the speaker-similarity gap. The reported similarity diagnostics therefore distinguish text-score selection from the remaining timbre cost addressed by allocation.

D.1 Calibration-Set Selection and Speaker Similarity

  • The calibration-set comparison uses 32-item snapshot sets at steps {0, 14}, with task-best selected by summed CER/WER and lossmax/lossmin by initialization reconstruction loss.SIM is a post-hoc diagnostic, and the sets are item-disjoint from Seed-TTS-eval.Source: §D.1 · p. 13
  • Lossmax had the highest development text score at both scales but a somewhat larger SIM decrease than task-best, while SIM changes were uniformly negative.SIM was not used by either calibration-set selection rule; all four reported calibration sets were slightly below Full Prec. at both scales.Source: §D.1 · p. 13
  • Late-FFN allocation separately targets the remaining speaker-similarity cost rather than retrospectively selecting calibration items on SIM.The comparison validates uniform-random, task-best, and lossmax on held-out remainders after applying the final late2 protocol.Source: §D.1 · p. 13

D.2 Detailed Snapshot-Placement Results

Snapshot placement is evaluated conditional on the scale-specific lossmax items, then held-out activation-restoration diagnostics test where restoring precision helps. Late sampler steps and FFNs show the strongest SIM gains among the tested restoration choices.

D.2 Detailed Snapshot-Placement Results

  • The snapshot study compares early2, late2, keep4, and keep8 allocations under a 64-snapshot budget, retaining 32, 32, 16, and 8 ranked lossmax items, respectively.The keep4 and keep8 sets are deterministic nested prefixes, and inference remains uniform W4A4.Source: §D.2 · pp. 13–14, Table S8 · p. 15
  • Snap-late2 exceeds the first+last reference on five of six subset×scale comparisons, with summed SIM gains of 0.0009/0.0014 at 1B/3.5B, and is adopted for held-out experiments.Snap-early2 decreases SIM on all three subsets; these results are conditional on the fixed lossmax items, not a universal placement optimum.Source: §D.2 · p. 14, Table S8 · p. 15

E Additional Allocation and Weight/Activation Diagnostics

  • Held-out restoration diagnostics change selected activation inputs from A4 to A32 while keeping W4 weights and other activations at A4, measuring changes against uniform W4A4.The all-A32 restoration arm corresponds to the W4/A32 corner of Table S10.Source: §E.1 · p. 15

E.1 Held-Out Activation-Restoration Diagnostics

  • At 1B/3.5B, late-window restoration gains +0.0043/+0.0034 summed ∆SIM versus −0.0013/−0.0002 for early restoration.FFN restoration also exceeds self-attention restoration: +0.0056 versus −0.0015 at 1B and +0.0027 versus −0.0005 at 3.5B.Source: §E.1 · p. 15
  • Late-FFN restoration has the largest summed ∆SIM per restored share at both scales, but these point-estimate rankings do not establish an optimal window or formal time×module interaction.It covers 16.67% of parameter-weighted activation sites at 1B and 15.79% at 3.5B, versus 100% for all-A32 restoration.Source: §E.1 · p. 15

E.2 Weight and Activation Corners

The weight-only and activation-only diagnostics use different construction procedures, so their differences cannot be interpreted as a symmetric factorial decomposition. Both remain close to Full Prec. on text metrics, while their SIM results vary by subset.

E.2 Weight and Activation Corners

  • Both single-factor configurations remain close to Full Prec. on text axes at both scales, while SIM is lower on zh and en at both scales.For W32A4 on hard, the SIM change is −0.0001 at 1B and +0.0003 at 3.5B.Source: §E.2 · p. 16
  • The two diagnostics are not symmetric factorial cells, so their differences cannot be added or read as weight×activation main effects or an interaction.W32A4 is independently calibrated; W4A32 disables activation quantization in the frozen W4A4 model.Source: §E.2 · pp. 15–16

F Deployment Protocol and Efficiency

The deployment comparison finds a memory reduction but a length-dependent latency tradeoff for the deployed uniform-W4A4 path. Difficulty-guided weight allocation does not yield a stable cross-scale rule for choosing blocks.

F Deployment Protocol and Efficiency

  • The deployment comparison uses matched FP32 and W4A4 export graphs, with W4A4 replacing DiT linear projections while other DiT operators and the text encoder remain FP32.The VAE runs in FP16; timings use batch-1 voice cloning across three fixed output lengths.Source: F Deployment Protocol and Efficiency · p. 16
  • On the medium input, peak memory is 5.90/2.82 GB at 1B and 15.83/4.44 GB at 3.5B for FP32/W4A4, while W4A4 is slower at five of six tested scale-length combinations.At 3.5B on the long input, W4A4 latency is 0.695× FP32, a 1.439× relative speedup; the path is an unfused deployment reference.Source: F Deployment Protocol and Efficiency · pp. 16–17, Table S11 · p. 17
  • At batch size 1, separate quantization kernels and dtype conversions dominate short-output overhead; the relative latency penalty falls as token count grows.The measurements cover three output lengths of an unfused reference and do not locate the crossover between medium and long inputs.Source: F Deployment Protocol and Efficiency · p. 17

G Alternative Repair Strategies

  • The alternative weight-precision test assigns W8 to six hardest or easiest blocks at matched average weight precision, while activations remain A4.The separate speaker-gradient channel-weighting intervention changes the reconstruction objective and is not part of final TTS-Q.Source: §G.1 · p. 18, G Alternative Repair Strategies · p. 18

G.1 Difficulty-Guided Weight-Precision Allocation

  • At 1B/3.5B, hardest-block allocation changes summed SIM by −0.0074/+0.0048, reversing direction across scales; English WER increases at both scales.The equal-budget easiest-block control also reverses relative SIM ranking across scales, so residual reconstruction difficulty provides no stable cross-scale precision rule.Source: §G.1 · p. 18, Table S13 · p. 18

G.2 Speaker-Gradient Channel Reweighting

Speaker-gradient channel reweighting changes per-block reconstruction weighting using final-step SIM gradients. It does not provide a stable cross-scale repair: text accuracy worsens, and summed SIM changes reverse direction by scale.

G.2 Speaker-Gradient Channel Reweighting

  • A fixed channel profile from averaged absolute final-step SIM gradients replaces uniform channel averaging in per-block reconstruction MSE and is applied to late2 snapshots.The gradients are averaged across token positions, lossmax items, and conditional and unconditional branches; normalization gives each block a mean channel weight of one.Source: §G.2 · pp. 18–19
  • 0.824 and 0.962 percentage-point summed CER/WER increases at 1B and 3.5B accompany summed SIM changes of +0.0046 and −0.0044, respectively.The 3.5B SIM decrease includes a 0.0064 decrease on hard.Source: §G.2 · p. 19
  • The experiments do not localize speaker information: neither reconstruction difficulty nor gradient magnitude identifies a transferable place to spend precision.The tested interventions do not establish that selected blocks or channels fail to encode speaker identity.Source: §G.2 · p. 19
Loading 2610.12474v1…