PaperlayerView exact revision on arXiv

Source-linked AI summary

GEPARD - Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue

Denis Pavlov, Ulanbek Abdurazakov, Nursultan Bakashov

arXiv:2609.04222v1eess.AScs.CLcs.LGcs.SD

TL;DR

GEPARD addresses the latency and serving constraints of interactive voice agents while avoiding specialized autoregressive decoders incompatible with standard vLLM serving. It combines modality-aligned text-audio training, targeted diagnostics for short-phrase collapse, and DPO-based guidance distillation, reducing failures while improving stability on long texts.

  • Problem

    Interactive voice agents require low time to first audio and scalable inference, while specialized autoregressive TTS decoders can conflict with standard vLLM serving constraints.

  • Method

    GEPARD uses modality-scale alignment, diagnostic probes for short-phrase runaway generation, and length-normalized DPO with CFG-positive preference pairs to distill guidance into model weights.

  • Results

    Single-pass clean generation reached 67.1% after three DPO rounds, while the full diagnostic failure rate fell by approximately 25 times and long-text derailment reached 0% across voices.

  • Takeaways & Limitations

    The distilled model makes long-text generation production-ready and nearly eliminates runaway behavior, although short-input WER remains sensitive to sampling variance and instability.

  • Takeaways & Limitations

    Voice cloning remains proof of concept because representation leakage produces low WavLM similarity, with SIM = 0.585 and no guarantee of full timbre transfer to unseen voices.

Abstract

from arXiv · show

We present GEPARD (Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue), a streaming text-to-speech model for real-time spoken dialogue. GEPARD generates speech autoregressively with an LLM backbone - text and audio embeddings are trained together in a single decoder-only model - and decodes it to a waveform with an FSQ-based neural codec, streaming audio chunk-by-chunk as text arrives. Our central goal is a TTS architecture served by a standard LLM engine (vLLM) without modifying its compute kernels. This defines the overarching design principle: the backbone is a standard full-attention transformer, while all non-trivial auxiliary mechanisms - zero-shot voice cloning, text augmentation, and classifier-free guidance - are moved out of the autoregressive decode loop into prefill, or distilled directly into the weights. On streaming end-to-end inference, a single stream reaches a Real-Time Factor of about 0.067 (roughly 15x faster than real-time); under 256 concurrent streams the system reaches an aggregate speedup of about 204x on a single server-class GPU. We detail: (1) system-level solutions for vLLM-native serving; (2) the "short register" (1-2 word) failure mode of autoregressive speech decoders, with diagnostic probes and a mitigation; and (3) distillation of two-pass classifier-free guidance over text into single-pass weights via Direct Preference Optimization (DPO).

1. Introduction and Problem Formulation

Gepard targets real-time spoken dialogue under a strict requirement to run on stock vLLM, preserving standard LLM serving while relocating auxiliary mechanisms outside autoregressive decoding. The paper investigates this design through a vLLM-native architecture, short-register diagnostics, CFG distillation, and multilingual scope limitations.

  • 1.1. Motivation and Central Thesis: Interactive voice agents require low time to first audio and low scaling cost, motivating TTS built on a standard vLLM engine.vLLM provides continuous batching and PagedAttention for standard LLM architectures.
  • 1.1. Motivation and Central Thesis: Custom decode-loop operations such as depth-transformers, intermediate cross-attention, or per-step two-pass CFG prevent stock vLLM serving.These operations break continuous batching and reduce system throughput.
  • 1.2. Overarching Design Principle: Gepard keeps a standard full-attention transformer and moves non-standard transformations to prefill, offline data generation, or fine-tuned weights.This is the paper’s overarching design principle for preserving serving-engine compatibility.
  • 1.2. Overarching Design Principle: Voice cloning uses a prefilling speaker prefix, short-phrase collapse uses text augmentation, and CFG is distilled into a single-pass model with offline paired generations.These mechanisms avoid dynamic changes inside the autoregressive decode loop.
  • 1.3. Inference Speed and Scaling: 0.067 RTF in production single-stream inference and approximately 204x aggregate speedup at 256 concurrent streams demonstrate the vLLM-native serving objective.The early uncontrolled sanity run measured approximately 0.040 RTF and 0.032 seconds TTFA, while the production measurement used end-to-end streaming under concurrency.
  • 1.4. Scientific and Practical Contributions: The paper studies 1–2-word runaway generation as a distinct failure mode using targeted diagnostics and a compensation method.The investigation focuses on entropy and stop-probability probes rather than relying only on aggregate benchmark WER.
  • 1.4. Scientific and Practical Contributions: DPO transfers improvements from two-pass CFG into a single-pass model using length- and quality-normalized reward scaling.This is presented as a contribution alongside the vLLM-native system and short-register analysis.
  • 1.5. Scope and Assumptions: The current system is an early baseline whose audio interface was optimized primarily on English data, leaving quality in other languages limited.The paper identifies multilingual alignment and targeted audio-interface fine-tuning as future work.

2. Architecture

Gepard is a decoder-only autoregressive TTS model that combines text, optional speaker-prefix, and audio streams in a stock transformer, then predicts neural-codec frames and speech termination. GroupFSQ enables parallel 32-channel frame generation compatible with standard vLLM, but makes training convergence harder.

  • Overview: Gepard takes text and generates discrete neural-codec audio codes, predicting only audio tokens and the end of speech.The decoder-only model has no text-generation head.
  • Overview: The architecture concatenates an optional Q-Former speaker prefix, text embeddings, and 32-channel audio inputs before a stock full-attention Qwen3.5 backbone.The prefix is removed before the 32 codebook heads and stop head operate on the audio region.
  • Backbone: The backbone remains standard full attention without internal hooks or decode-time custom operations, supporting compatibility with FlashAttention-2 and stock serving.LinearBlock layers were removed from Qwen3.5, and the remaining backbone was trained from scratch.
  • Codec: GroupFSQ replaces RVQ because its orthogonal channels have negligible conditional total correlation, allowing all 32 channels to be sampled in one autoregressive step.This avoids sequential codebook generation and auxiliary depth-transformers that complicate optimized LLM serving.
  • Codec: Frame-by-frame generation predicts 32 independent channels simultaneously, creating a dense high-entropy target that slows convergence and increases decoding uncertainty.The authors explicitly state that FSQ was selected for generation speed and serving compatibility, not training simplicity.
  • Codec: NanoCodec’s 8 packed tokens are unfolded into 32 channels with capacities {8, 7, 6, 6}, enabling 216 total output logits instead of 16,128.The mixed-radix representation supports 32 independent tiny classification heads.
  • Voice Cloning: Unconditional exposure combines approximately 3.05% sentinel rows with stochastic dropout, ensuring the dense speaker bucket follows the CFG-unconditional path.This exposure is required for inference-time CFG to function.
  • Voice Cloning: The voice compressor can transfer timbre without accent or language, but this observation is qualitative and consistent with a content-invariant 8-token speaker representation.A Russian reference can produce fluent English without a Russian accent, and the same reference works across several languages.

3. Engineering Discoveries Verified by Experiment

The engineering framework addresses modality mismatch, codebook factorization, and stopping behavior in decoder-only TTS through targeted architectural choices and diagnostics. Experiments show stable adaptation without representational collapse, support parallel GroupFSQ sampling, and identify late stopping as a backbone-linked failure mode.

  • Modality Scale Alignment Framework: Direct MLP coupling removed a gradient barrier that made audio gradients ≈2400 times smaller than text gradients, increasing audio-interface gradients by 10–20 times.The earlier Linear → RMSNorm → ×0.02 path prevented the audio components from training effectively.
  • Modality Scale Alignment Framework: Text representations adapted without collapse: drift stabilized at ≈0.34, cosine similarity remained 0.948, and effective rank stayed ≈955/1024.The gradient norm smoothly decreased to a baseline of ≈1.5 while total loss converged without singularities.
  • Modality Scale Alignment Framework: Audio lookup tables retained ≈96–97% of their structural rank ceiling across all 32 channels, indicating full capacity utilization and no intra-head collapse.The ceiling is determined by each channel’s number of FSQ codes.
  • GroupFSQ and Factorized Sampling without Depth-Transformer: GroupFSQ channels are independent by design, making conditional multi-information approximately zero and factorized parallel sampling correct without a depth-transformer.External Magpie-TTS tests support successful synthesis without its optional local transformer, while a direct ablation in GEPARD remains future work.
  • Stop Head as a Bernoulli Predictor: Class Imbalance and Saturation: GEPARD models speech termination with a separate Bernoulli stop head and independently samples 32 audio channels at each step.The stop likelihood treats natural termination separately from trajectories truncated at the maximum frame limit.
  • Stop Head as a Bernoulli Predictor: Class Imbalance and Saturation: Positive weighting of 25.0 compensates for stop-event imbalance, after which 99.7% of non-stop frames have p_stop,t between 0.0001 and 0.004 and terminal frames reach 1.0.Without reweighting, standard BCE converges near p_stop,t≈0.05 and fails to cross the 0.5 inference threshold.
  • Stop Head as a Bernoulli Predictor: Class Imbalance and Saturation: Late-stop failures localize short-input runaway behavior to the backbone rather than the stop head, which eventually outputs 1.0 on self-generated states.The delayed trigger occurs a few seconds after generation should have ended.

4. Data and Training

GEPARD uses staged training and targeted data augmentation to support multilingual speech generation, while acknowledging that its English-heavy data limits other-language quality. Pretraining provides an acceptable base, but fine-tuning is needed to improve intelligibility and address short-input failures.

  • Training strategy: Training proceeds through pretraining, SFT, and DPO, with DPO specifically addressing the short-register failure mode.DPO is introduced after SFT, while training stability is handled separately.
  • Data: 27,623,833 samples spanning 68,833 hours and 1,675,752 speakers form the multilingual pretraining mix, whose clips are concentrated in the 3–10 second range.The corpus includes 19 sources and has a 21.5 Hz frame rate.
  • Data provenance and balance: English dominates the corpus, so the multilingual backbone has stronger English audio quality while other languages remain limited by data volume.The authors attribute this imbalance to the predominance of high-quality open English TTS corpora.
  • Short-input mitigation: Text repetition augments short inputs because a fixed 8-token prefix can dominate one- or two-token prompts and prevent the model from locking onto speech variety.Only the final repeated copy triggers audio rendering, while context copies are masked from supervision.
  • SFT stage: 19–64% lower WER and 45–55% lower CER across speakers followed LoRA-SFT, but MOS remained unchanged and short-input hallucinations persisted.The results indicate improved intelligibility without eliminating the short-input failure mode.
  • SFT stage: ≈27% successful generations on the most challenging slice remained under clean single-pass inference after improved SFT, motivating preference optimization.CFG showed steady improvement, but production-style single-pass inference still failed frequently.

5. Short Register as a Failure Mode

The short-register failure affects 1–2-word inputs systemically: autoregressive generation often runs away because the speaker prefix overwhelms weak text conditioning. Diagnostics localize the problem to backbone convergence rather than phonetics, stop-head blindness, or speaker coverage, motivating text repetition and DPO-based CFG distillation.

  • Failure mode: 1–2-word inputs trigger systemic runaway generation, with diagnostics designed to identify the structural failure mode rather than rely on aggregate WER.The study uses targeted probes to rule out competing explanations.
  • Severity: 91.5% of baseline SFT generations failed, including 36.7% that timed out and a median WER of 2.0 from repeated output.Failure rates ranged from 86.8% to 98.5% across voices, indicating the defect was not tied to particular words.
  • Diagnostic probes: Stop probabilities were effectively binary, while late-stop trajectories showed the head can fire after the backbone exits its speaking mode.Among 80 runs, 28 were late-stop and 31 never-stop; 31 of 59 runaway trajectories would not have been fixed by stop-sampling.
  • Diagnostic probes: Derailing trajectories maintained high entropy near 1.5, while clean trajectories converged from 1.53 to 0.82; lowering onset temperature did not solve the failures.The small initial NLL difference and zero WER on converged trajectories argue against a single bad frame or phonetic confusion.
  • Structural cause: The structural cause is prefix dominance: eight speaker tokens overwhelm 1–2 text tokens, spreading audio attention across the prefix and weakening text conditioning.This produces high-entropy tokens and failure to lock on in the decoder-only architecture.
  • Mitigation: 6.5× higher success on challenging voice F2 validates increasing text conditioning through repetition and CFG-generated DPO positives.Text repetition reduced failure from 18.8% to 5.0%, while the stability plateau began at 13–15 tokens, motivating a target of approximately 16.
  • Mitigation: 67.1% clean single-pass generation after DPO Round 3 exceeded the improved SFT CFG ceiling of 64.8%, while the full diagnostic failure rate fell by approximately 25 times.DPO-R2 was selected as the final configuration because Round 3 introduced undesirable OOD drift.
  • Limitations: DPO drifted on out-of-distribution prompts because the KL anchor constrained only the distribution of preference pairs.The reported mitigation is broader data coverage rather than reward modification.

6. Inference and Evaluation

Gepard is evaluated as a vLLM-native, single-pass streaming TTS system for speed and voice-cloning quality. It achieves substantial serving throughput, but speaker similarity remains its main quality weakness and the benchmark under-tests the short-register failure mode.

  • Inference design: The production path uses standard vLLM with a full-attention transformer, moving speaker conditioning and other non-standard processing outside autoregressive decoding.The Q-Former prefix is computed during prefill, while single-pass inference is used for serving.
  • Speed: 203.9× peak aggregate speedup is reached at up to 256 concurrent streams, while the practical interactive operating range is approximately 64–128 streams.At 256 streams, per-stream RTF rises above real-time, making that setting more suitable for latency-tolerant batch processing.
  • Speed: 0.067 RTF is achieved for a single stream, approximately 15× faster than real-time.The reported production measurement is end-to-end and includes the backbone and neural codec.
  • Quality: Gepard leads the voice-cloning cohort on NISQA-MOS and interpretable signal-quality dimensions, but its SIM of 0.585 is the cohort minimum.The results identify speaker similarity, rather than signal quality, as the main weakness.
  • Quality: WER 0.036 is lower-middle in the voice-cloning cohort, whereas the benchmark contains almost no 1–2 word prompts and therefore under-tests the project’s primary short-register failure mode.The short-register adversarial slice and Seed-TTS-eval refer to different evaluation sets.
  • Quality: The reported naturalness observation is qualitative: the model exhibits realistic intonation, breathing, and rhythm, which the listed objective metrics do not directly capture.NISQA-MOS 4.25 is presented as only a partial proxy for liveliness.

7. Limitations and Future Work

The current version’s principal limitation is voice-cloning representation leakage, which leaves speaker similarity weak and cloning at proof-of-concept status. Future work focuses on better compressor training, controlled ablations, and broader language coverage.

  • Limitations: SIM = 0.585 is the most critical limitation because GroupFSQ and the Q-Former can encode reference spectral and acoustic characteristics instead of speaker timbre.SupCon regularization partially addresses leakage but does not guarantee full timbre transfer to unseen voices.
  • Limitations: The voice-cloning mode remains a Proof of Concept because representation leakage is not fully resolved for unseen voices.The limitation is explicitly tied to the compressor’s training dynamics and incomplete timbre transfer.
  • Future work: Planned work includes training the compressor on unrelated same-speaker reference and target pairs and expanding evaluation to unseen speakers.These changes are intended to address leakage and strengthen voice-cloning validation.
  • Future work: The study also plans systematic ablations of the backbone and depth-head projector to quantify their respective contributions.The proposed comparisons target architectural choices that were not yet validated systematically.
  • Future work: Expanding non-English training data and targeted audio-interface fine-tuning is planned to improve quality parity across supported languages.The current audio interface was optimized primarily on English data.

8. Release and Reproducibility

The release provides a parameter chain from base pretraining through LoRA-SFT and DPO, together with the codec, serving configuration, and reproducibility-oriented implementation details. DPO-r2 is selected as the final model because it balances stability and quality better than the third round.

  • Release artifact: The released parameter chain consists of a 555.7M-parameter base model, LoRA-SFT, and a final DPO model.The SFT stage uses text repetition and length reweighting, while DPO produces the final preference-optimized model.
  • Release artifact: DPO-r2 is selected as the final configuration because DPO-r3 introduces undesirable drift on out-of-distribution prompts.The second round is described as preserving the best balance of stability and quality.
  • Release artifact: The neural codec is nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps, with an optional deterministic frame limit for inference.The release also includes model parameters, data-license specifications, a demo, and the optimized serving solution.
  • Configuration: The implementation specifies a 32-head audio output with dimensions [8, 7, 6, 6] × 8 and a stop-loss positive-class weight of 25.0.These settings define the codec-head and stop-predictor configuration documented for reproducibility.
  • Configuration: The documented data and conditioning configuration uses NanoCodec at 21.5 frames/s, active text repetition, and K=8 latent queries with L=2 compressor layers.The compressor is frozen during fine-tuning.

Appendix A. Mathematical Description and Implementation Details

The appendix formalizes Gepard’s codec representation, audio prediction losses, voice compressor, regularizers, and preference-optimization objectives. These details specify how representations are unfolded, trained, normalized, and ranked during implementation.

  • Codec representation: The codec unfolds packed tokens into four independent FSQ channels, and eight codebooks yield 32 independent channels with cyclic dimensions [8, 7, 6, 6].The mixed-radix representation is dequantized channel-wise without residual dependencies.
  • Audio interface: Audio embeddings are initialized with Gaussian weights, scaled to the embedding-token standard deviation, and aligned after discarding the prefix hidden states.The alignment maps backbone hidden states to the temporal audio region.
  • Losses: Training combines causal-shifted cross-entropy over 32 encoding heads with weighted binary cross-entropy for the stop head.Padding is excluded from valid channel indices, and the stop positive class uses weight 25.0.
  • Voice compressor: The voice compressor uses K=8 learnable queries and L=2 pre-norm blocks, with CFG dropout replacing the prefix by null_prefix at probability 0.15 or for null sentinels.The compressor’s inference output is the decoder prefix, while regularizers operate on normalized query representations during training.
  • Voice compressor: Diversity and supervised contrastive regularization are computed from normalized query representations before curriculum masking, with SupCon using temperature τ=0.1.Positive examples are same-speaker samples in the batch, with negatives expanded through cross-rank merging.
  • Preference optimization: The length-normalized Bradley–Terry objective compares selected and rejected trajectories, while automatic rewards combine duration bounds, WER, and cosine similarity.Trajectory likelihood includes the 32 audio heads and Bernoulli stop-probability components with a stop-probability floor of 10^-4.

A.7. Speed Metrics

This section defines the speed metrics and evaluation setup for streaming inference. It distinguishes first-audio latency from duration-based throughput and specifies a burst-concurrency scenario measured locally.

  • Speed calculations use each request’s wall duration, generated audio length, and total elapsed time for the concurrent batch.
  • Streams generating less than 2.0 s of audio are filtered out to prevent stochastic early stops from biasing percentiles.
  • Concurrency tests use localhost loopback and an instantaneous burst of concurrently initiated streams, modeling worst-case prefill contention rather than Poisson arrivals.
  • TTFB measures wall time elapsed until the first audio delta, equivalently TTFA.
  • The reported architectural and optimization choices were derived from targeted empirical diagnostics during development.

B.1. Modality Scale Mismatch

Early training exposed a severe scale mismatch between text and audio representations: audio dominated attention, weakening text conditioning. Subsequent changes removed the resulting gradient bottleneck.

  • A 60× scale mismatch caused audio tokens to dominate self-attention, leaving text tokens with approximately 50× less attention weight.
  • The attention imbalance caused the model to ignore text context and optimize as an audio-only autoregressive generator.
  • Scaling audio embeddings by s_audio = 0.02 aligned forward-pass scales but attenuated backward gradients by 50×.
  • RMSNorm and projection effects further attenuated and homogenized gradients, while the combined gradient reaching audio tables was approximately 2400 times weaker than in backbone hidden states.
  • Later iterations removed the intermediate RMSNorm, moved the scaling factor, and initialized audio tables with pretrained text embedding variance.

B.3. Representational Alignment in Hidden Space

The diagnostics distinguish representational capacity from generation instability and show that the full-attention backbone aligns text and audio progressively in deeper layers. Rollout evidence indicates that short-phrase failures reflect variance rather than inability to synthesize the words.

  • Representational alignment: Input text and audio embedding matrices remained nearly orthogonal, with cosine similarity close to zero.
  • Representational alignment: Hidden-state cosine similarity increased from −0.04 at Layer 0 to 0.98–0.99 at Layers 12 and 13, indicating deep cross-modal alignment.
  • Short-phrase failures: The short-phrase diagnostic tested whether high WER reflected limited synthesis capacity or high generation variance.
  • Short-phrase failures: In 100% of failure-prone rollout groups, the minimum WER across 20 rollouts was 0.00, showing that every stress-test word could be synthesized.
  • Short-phrase failures: These rollouts supported preference optimization through DPO as the appropriate lever for the observed short-phrase failures.

Appendix C. Comparison Against Commercial and Large-Scale Systems

The appendix situates Gepard against commercial and large-scale systems on a shared 1088-prompt evaluation, while separating the comparison from voice-cloning results. Commercial systems define the upper envelope, and Gepard remains the strongest open-source voice-cloning entry on NISQA-MOS in the field view.

  • Evaluation scope: Five commercial or large-scale systems were evaluated on the same 1088 Seed-TTS-eval prompts using intelligibility and signal-quality metrics without reference cloning.
  • Evaluation scope: The comparison omits SIM because it is undefined in non-cloning mode and is not intended to claim voice-cloning parity.
  • Results: Commercial systems form a high-quality cluster with WER ≈0.014–0.017, zero hard failures, and the field’s highest NISQA-MOS values.
  • Results: Gepard’s WER is 0.036, below the commercial cluster but above VibeVoice’s 0.109, while its NISQA-MOS is 4.25.
  • Results: Across all 12 models, Gepard is mid-field on WER and the best open-source voice-cloning model on NISQA-MOS.
Loading 2609.04222v1…