Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning

1 Introduction

• Voice cloning faces a gap between high-quality reproduction and systems that are efficient, transcript-free, and free of upfront duration constraints.Autoregressive semantic decoding adds latency, while non-autoregressive systems often require reference transcripts and fixed-length decoding requires duration in advance.Source: §1 · pp. 1–2
• Training-free length control estimates speaking pace from reference audio and combines it with the target text’s syllable count for transcript-free decoding.The approach also distills discrete semantic codes and continuous latent representations without access to the teacher’s original large-scale training corpus.Source: §1 · p. 2
• Tacit-TTS achieves competitive zero-shot quality, runs over 10× faster than IndexTTS2 for utterances longer than 5 seconds, and supports cross-lingual and non-lexical references.Transcript-dependent systems often degrade or fail with these references because their ASR transcripts are unreliable.Source: §1 · p. 2
• Tacit-TTS evaluates cross-lingual references from eight languages and non-lexical references including infant babble and synthetic gibberish.Transcript-dependent systems often degrade or fail in these settings because ASR transcripts are unreliable.Source: §1 · p. 2
• Tacit-TTS replaces IndexTTS2’s autoregressive text-to-semantic stage with masked non-autoregressive generation, accelerating that stage by 26.5× and using ReFlow distillation for the renderer.The masked generator is trained on teacher-synthesized data while retaining pretrained speaker and emotion conditioning.Source: §1 · p. 2

2 Related Work

• Masked prediction generates tokens in parallel through iterative unmasking; MaskGCT applies this scheme to zero-shot TTS, while Tacit-TTS distills it from an autoregressive teacher [Wang et al., 2025b].IndexTTS2 conditions its autoregressive model on w2v-BERT features compressed into speaker and emotion embeddings [Zhou et al., 2026b; Deng et al., 2025].Source: §2 · p. 2
• IndexTTS2’s flow-matching renderer requires roughly 25 Euler steps; Tacit-TTS fine-tunes it on its text-to-semantic model’s outputs and uses ReFlow to integrate straightened trajectories in 4 to 8 steps.The paper attributes the resulting efficiency to this renderer approach together with the masked decoder.Source: §2 · p. 3
• Many zero-shot systems require reference transcripts, so unsupported languages or non-lexical speech can leave them conditioning on empty or hallucinated text.IndexTTS2 removes this dependence by conditioning on self-supervised w2v-BERT representations compressed by a Perceiver; the paper reports Tacit-TTS remains effective with cross-lingual and non-lexical references.Source: §2 · p. 3

3 Method

• Tacit-TTS uses a two-stage pipeline: masked-LM generation predicts semantic tokens from reference audio and target text, then a semantic-to-mel model and vocoder synthesize speech.The system retains IndexTTS2’s speaker and emotion conditioning while replacing its autoregressive semantic decoder.Source: §3 · p. 3, Figure 2 · p. 3

3.1 Text-to-Semantic Generation

• A bidirectional non-autoregressive Transformer predicts masked semantic tokens in parallel, committing confident predictions over a fixed number of iterations independent of output length.Text embeddings are upsampled to the target semantic length and fused with token embeddings, keeping the Transformer sequence length at that target length.Source: §3.1 · pp. 3–4

3.1.1 Training-Free Length Control

• Target semantic length is estimated without a reference transcript by combining target-text syllable count with speaking pace estimated from the reference waveform.The method estimates acoustic syllables from prominent energy peaks in voiced regions, clamps the pace estimate, and enforces a minimum length of 8.Source: §3.1.1 · p. 4

3.1.2 Text-to-Semantic Distillation

• T2S distillation trains on teacher-synthesized examples and jointly predicts discrete semantic codes and recovers continuous residual representations for the downstream renderer.Reference voices are paired with content-independent target text; the two heads share a Transformer trunk.Source: §3.1.2 · pp. 4–5

3.2 Semantic-to-Audio Generation

• The semantic-to-audio stage retains IndexTTS2’s conditional flow-matching renderer and BigVGAN vocoder, reducing renderer inference from about 25 Euler steps to 4–8 through reflow distillation.The renderer is first fine-tuned for the T2S model’s conditioning distribution, then trained on noise-to-output couplings recorded from that renderer.Source: §3.2 · p. 5, §3.2.1 · p. 5

3.2.1 Semantic-to-Mel Reflow

• After reflow, the renderer achieves comparable synthesis quality using 4–8 Euler steps instead of approximately 25, with the flow loss covering only non-prompt mel frames.Reflow uses noise–output pairs from the fine-tuned renderer to encourage straighter generation trajectories.Source: §3.2.1 · p. 5

3.3 Training Data

• Distillation uses separate datasets: 874k T2S samples and 8k renderer couplings, against IndexTTS2’s 55k hours of training data; the authors report comparable English speaker similarity despite the smaller budget.The T2S set covers about 2k hours and 2,739 reference speakers; the renderer coupling set corresponds to about 36 hours.Source: §3.3 · pp. 5–6

4.1 Experiment Setup

• Evaluation covers four full zero-shot test sets and additional references in eight languages, infant babble, and synthetic gibberish, using IndexTTS2 and four open systems as baselines.The standard test sets pair each reference clip with a content-independent target sentence; quality baseline results come from the IndexTTS2 paper, while cross-lingual and non-lexical comparisons were run by the authors.Source: §4.1 · p. 6

4.2 Main Results

• Tacit-TTS has the best English speaker similarity among deployable systems and competitive content accuracy across four English and Mandarin test sets.The authors report perceptual quality close to IndexTTS2 across all four sets; the weaker Mandarin results may partly reflect unequal speaker coverage.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 Cloning from Cross-Lingual References

• Across eight cross-lingual reference languages, Tacit-TTS reaches 0.789 SS with 0.10% WER on English targets and 0.692 SS with 1.90% CER on Chinese targets.Its error rate stays below 3.4% across all eight languages; transcript-dependent baselines sometimes produce unrelated or repeated content or fail.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Cloning from Non-Lexical References

• Both transcript-free systems generate speech for all infant-babble and synthetic-gibberish references, while transcript-dependent baselines can be unstable or fail.Tacit-TTS has higher speaker similarity than IndexTTS2 on infant babble for both English and Chinese targets; both systems perform well on synthetic gibberish.Source: §4.4 · p. 8, Table 3 · p. 8

4.5 Efficiency Analysis

• T2S latency falls from 5274 to 199 ms (26.5×), and generation latency from 6149 to 509 ms (12.1×).Reference conditioning is an unchanged one-time cost; ReFlow reduces S2A latency from 875 to 310 ms.Source: Table 4 · p. 8, §4.5 · p. 9
• Speedup over IndexTTS2 rises from 9.0× at four seconds to 14.9× around thirty seconds, then narrows to 11.8× at 125 seconds.The figure plots end-to-end generation time against output duration, including reference conditioning; the authors attribute the later narrowing to full self-attention costs.Source: §4.5 · p. 9, Figure 3 · p. 9
• Twelve T2S unmasking steps and eight S2Mel steps are the default: more steps improve content accuracy and speaker similarity, respectively, with limited further content gains beyond 12 T2S steps.The authors selected these settings through a grid search.Source: §4.5 · p. 9

4.6 Duration Estimation

• The training-free, transcript-free length estimator produces durations whose correlation with ground truth is comparable to other systems.The evaluation uses Pearson correlation because natural speech duration varies even for the same utterance.Source: §4.6 · p. 9

5 Conclusion

• The conclusion presents Tacit-TTS as a transcript-free, efficient zero-shot voice-cloning system distilled from IndexTTS2.Its approach combines masked non-autoregressive semantic generation, training-free length control, and an accelerated flow-matching renderer.Source: §5 · pp. 9–10
• Across two English and two Mandarin datasets, Tacit-TTS achieves competitive zero-shot quality at a fraction of the teacher’s generation cost.The conclusion also reports effectiveness with cross-lingual and non-lexical references.Source: §5 · p. 10

Ethics Statement

• Transcript-free cloning lowers the barrier to using arbitrary recordings for impersonation, fraud, or non-consensual synthetic speech.The authors recommend requiring speaker consent and pairing deployment with audio watermarking and synthetic-speech detection.Source: Ethics Statement · p. 10
• The work targets efficiency in an existing public system, not an expansion of what voice-cloning systems can clone.The authors used publicly released research datasets, collected no new recordings, and did not attempt speaker identification.Source: Ethics Statement · p. 10
• The authors report using publicly released datasets under their respective licenses and collecting no new recordings.They also state that they made no attempt to identify any speaker.Source: Ethics Statement · p. 10

Language Model Usage Statement

• The manuscript used a large language model only for grammar checking and minor language polishing; the authors reviewed and edited its suggestions.The authors state that the scientific content, system design, experiments, analysis, and conclusions are their work.Source: Language Model Usage Statement · p. 10

A Speaker and Emotion Conditioning Encoders

• The frozen speaker and emotion encoders reuse IndexTTS2 encoders and start from the same w2v-BERT-2.0 features.The speaker encoder pools features into 32 latent tokens, while the emotion encoder uses a single latent query.Source: A Speaker and Emotion Conditioning… · p. 14

B Non-Lexical Reference Set Construction

• Three infant-babble references are selected from 70 recordings by filtering voiced segments and choosing examples nearest three embedding clusters.The recordings contain pre-linguistic vocalizations without accompanying text; segments shorter than 3 seconds are dropped.Source: B Non-Lexical Reference Set Construction · p. 14
• Synthetic-gibberish prompts comprise 10 pronounceable pseudo-word sequences, each made from 1–3-syllable words and lacking real words or a valid transcript.Each sequence is paired with a reference voice utterance to generate the prompt audio.Source: B Non-Lexical Reference Set Construction · p. 14
• Table B.1 lists the 10 synthetic-gibberish text prompts.Source: Table B.1 · p. 14

C Perceptual Quality Evaluation

• Tacit-TTS remains close to IndexTTS2 on UTMOS and DNSMOS across all four test sets.IndexTTS2 is the distillation teacher and serves as the main reference, not a target to surpass.Source: C Perceptual Quality Evaluation · p. 15, Table C.1 · p. 15
• UTMOS and DNSMOS are used as reference measures of perceptual quality, not definitive rankings.Some systems score higher than ground-truth recordings, suggesting the predictors may favor speech distributions resembling their training data.Source: C Perceptual Quality Evaluation · p. 15
• Table C.1 reports zero-shot UTMOS and DNSMOS performance across the four datasets.Source: Table C.1 · p. 15

D Generation Step Ablation

• Table D.1 compares SS, WER/CER, and efficiency across T2S/S2Mel sampling steps on LibriSpeech test-clean.The test set has a mean duration of 7.1 seconds.Source: Table D.1 · p. 15
• The gray row marks the default configuration.Source: Table D.1 · p. 15
• Efficiency is reported as 1/RTF, with 1× denoting real time and one-time conditioning computation excluded.Source: Table D.1 · p. 15

E Training Data Construction

• Each T2S sample pairs reference speech with a content-independent target sentence matched to reference duration using corpus-specific speaking rates.Samples are generated in one IndexTTS2 teacher pass without waveform synthesis; the stored items include semantic codes, continuous latents, conditioning latents, and text tokens.Source: E Training Data Construction · pp. 15–16
• Training targets combine LibriSpeech and AISHELL text with generated commentary, rare-word supplements, ad-style lines, and extreme-length sentences.The voice pool draws from LibriSpeech, AISHELL-1, and ESD training speakers, excluding the four evaluation voices.Source: E Training Data Construction · p. 16
• The final dataset contains 874,000 samples, about 1,960 hours of semantic-code speech, and 2,739 unique reference voices.Oversampling the rare-word and extreme-length subsets yields 2,090 hours per epoch.Source: E Training Data Construction · p. 16
• The ReFlow coupling set contains 8,000 utterances pairing recorded Gaussian noise with final mels generated by the trained T2S model and fine-tuned renderer.The set uses 400 LibriSpeech and AISHELL-1 speakers with an equal English/Chinese split.Source: E Training Data Construction · p. 16

F Duration Fidelity Details

• Duration fidelity is measured with Pearson’s r between generated and ground-truth durations, pooled across all four test sets and jointly across English and Mandarin.Figure F.1 presents scatter plots and per-test-set results; each panel reports pooled r for one model.Source: F Duration Fidelity Details · p. 16, Figure F.1 · p. 16

• The reference set uses three 6–12-second clips per language across eight languages, with 12 male and 12 female speakers.The languages are not synthesis languages of Tacit-TTS or its teacher, which generate Chinese and English only.Source: G Cloning From Multi-lingual References · p. 17
• Table G.1 reports per-language results for six methods on English and Chinese targets, including failure rate and speaker similarity and error-rate measures.The caption defines Fail as the percentage of utterances with no generated speech and marks languages with no generated speech using ×.Source: Table G.1 · p. 17
• References are selected using speaker embeddings and clustering, with a cosine-similarity threshold to limit similarity between same-language clips.The selection procedure uses k-means on CAM++ embeddings and requires cosine similarity below 0.6 between any two references of the same language.Source: G Cloning From Multi-lingual References · pp. 17–18
• Transcript-dependent baselines use Whisper-generated reference transcripts, with MaskGCT also given a detected prompt language when supported.For unsupported languages, MaskGCT is given English, the only option available to a user.Source: G Cloning From Multi-lingual References · p. 18