Source-linked AI summary
TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context
Fritz Cremer, Jonathan Cremer
TL;DR
TontaubeV1 addresses the trade-off between natural prosody and efficient, low-latency text-to-speech. It uses hierarchical DualCodec prediction with independently sized transformers, bounded-context positioning, and causal reconstruction for streaming. On English audiobook prosody, it matches ElevenLabs Flash v2.5 and outperforms three listed comparators, while reported serving measurements reach 200 ms to first audio and an RTF of 0.08 for one input.
Problem
Text-to-speech systems trade natural prosody against computational cost, memory use, and latency, especially for lower-cost and smaller systems.
Method
TontaubeV1 predicts a semantic DualCodec stream with a large transformer, adds acoustic refinements with three smaller transformers, and uses bounded positions plus VibeVoice causal decoding for streaming.
Results
TontaubeV1 matches ElevenLabs Flash v2.5 and outperforms Fish Audio S2 Pro, the April 2026 Gradium API, and Cartesia Sonic 3 on English audiobook prosody.
Takeaways & Limitations
The system provides a single-consumer-GPU streaming path with approximately 200 ms to first encoded audio and an end-to-end RTF of 0.08 for one input.
Takeaways & Limitations
The evaluation establishes English reading prosody and word-level correctness, but not German or broader multilingual quality, voice similarity, conversational performance, long-form continuity, or streaming quality.
Abstract
from arXiv · showhide
Text-to-speech systems often face a trade-off between natural prosody and efficient inference: higher perceptual quality typically comes at increased computational cost and latency. We present TontaubeV1, a model that preserves natural prosody while enabling streaming from a single consumer GPU. Speech is encoded by the hierarchical DualCodec representation at 12.5 Hz, which separates a semantic stream from successive acoustic refinements. Our design assumes that prosodic structure is largely established when the semantic stream is generated, and allocates capacity accordingly: a Qwen3-1.7B-derived transformer predicts that stream and thereby the utterance duration, while three progressively smaller Qwen3-0.6B-derived transformers each add one acoustic refinement. Text is tokenized per character rather than by subword. Paired text and audio markers at shared positions support long-form generation with bounded context, and overlapping DualCodec reconstructions are mapped into the VibeVoice acoustic latent space and decoded causally, enabling streaming despite DualCodec's noncausal decoder. The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German, with additional multilingual support. The four predictors total 2.9B parameters; on a single RTX 5090 the streaming path reaches approximately 200 ms to first audio. In separate non-streaming measurements, the end-to-end real-time factor (RTF) is 0.08 for one input and the aggregate RTF is 0.02 across eight concurrent inputs. On our LLM-as-a-judge audiobook-reading benchmark, TontaubeV1 matches ElevenLabs Flash v2.5 and outperforms Fish Audio S2 Pro, the April 2026 Gradium API, and Cartesia Sonic 3 on prosody. The model weights are released on Hugging Face under the Tontaube Community Model License 1.0.
1 Introduction
TontaubeV1 targets lower-cost, single-GPU text-to-speech while preserving competitive prosodic quality. It combines hierarchical codec prediction with causal streaming reconstruction and evaluates naturalness on continuous audiobook reading.
- 2.9B total parameters let TontaubeV1 run and stream on a single consumer GPU while targeting lower-cost speech synthesis.
- At 12.5 Hz, DualCodec separates a semantic first codebook carrying content and timing from higher-index acoustic refinements.One minute of audio occupies 750 tokens at this frame rate.
- A Qwen3-1.7B-derived transformer predicts the semantic stream, whose output carries intonation and determines utterance duration; three smaller transformers add acoustic refinements.
- Overlapping DualCodec reconstructions are mapped into VibeVoice acoustic latents and causally decoded, allowing stable audio before generation finishes.
- On 400 English audiobook passages, TontaubeV1 matches ElevenLabs Flash v2.5 on prosody and outperforms Fish Audio S2 Pro, Gradium, and Cartesia Sonic 3.The evaluation targets monotony, misplaced emphasis, and incorrect phrase boundaries in continuous reading.
2 Related Work
TontaubeV1 follows language-model-based codec generation but factorizes codebooks into independently sized models. Its character-level tokenization, shared position timeline, bounded context, and streaming reconstruction distinguish the design.
- TontaubeV1 shares the language-model-plus-discrete-codec approach used by Fish-Speech and Qwen3-TTS, including a 12.5 Hz semantic-first tokenizer.
- Unlike interleaved single-pass systems or shared residual modules, TontaubeV1 assigns a separate model to each codebook and sizes them independently.Early experiments reported better convergence with separate models.
- Separate codebook models require four checkpoints and sequential within-chunk execution, but acoustic stages can be scheduled independently and carry no state across chunk boundaries.
- Character-level tokenization, occurrence-based rotary positions, and paired boundary markers align text and audio on one timeline while keeping transformer context bounded across long passages.
- Streaming reconstruction re-encodes overlapping DualCodec outputs into VibeVoice acoustic latents for causal decoding rather than training a causal codec.
3 Model Architecture
TontaubeV1 generates speech as four coarse-to-fine codec streams: one semantic stream fixes content and duration, while three acoustic stages add residual detail. Its architecture uses stage-specific conditioning, character-level spoken-text inputs, and length-matched acoustic refinement.
- Speech representation: TontaubeV1 models speech as autoregressively generated discrete codec streams conditioned on spoken-form text and optional reference-audio streams.The retained codec stack comprises one semantic stream and three acoustic streams.
- Speech representation: The retained representation uses one semantic stream and three acoustic streams, while later codec refinements are omitted from the model output.DualCodec provides eight streams at 12.5 Hz; retaining four yields 625 bit/s rather than 1,225 bit/s for the full stack.
- Four-stage codec generation: Each stage uses its own model and a restricted conditioning structure in which stage i receives only prompt streams P 0:i and previously completed streams C<i.This factorization imposes a coarse-to-fine inductive bias rather than changing the exact chain-rule factorization.
- Four-stage codec generation: The semantic stage generates C0 and determines the shared timeline, while CB1–CB3 each generate exactly L acoustic tokens for progressively finer streams.Later stages condition on the text, available prompt streams, and completed lower streams without revising C0 or its length.
- Input and output: Spoken text is tokenized one character at a time, whereas control syntax uses inherited subword pieces and dedicated structural tokens.The documented interface includes language and style controls, split markers, padding, and stage-specific codec vocabularies.
- Input and output: Optional reference audio is encoded into aligned prompt streams, and the acoustic stages are length-matched to C0 rather than emitting independent structural terminators.Prompt streams are jointly truncated to stage-specific frame limits before serialization.
4 Positions and Long-Form Layout
TontaubeV1 assigns positions by logical time rather than serialized order, aligning text, prompts, and codec streams. Paired chunk boundaries and rolling context preserve this timeline while keeping inference context bounded for long passages.
- 4.1 Positions: Logical-time positions align corresponding text and audio frames even when serialization places them far apart.The scheme assigns positions by when tokens occur rather than where they appear in the flat sequence.
- 4.1 Positions: Frame t receives position ℓP + t across rows, encoding simultaneity independently of serialized layout.Shared coordinates give corresponding streams zero relative positional offset under RoPE.
- 4.1 Positions: Prompt streams overlay one another, so four streams of length ℓP occupy ℓP coordinates rather than 4ℓP.Text and audio begin at ℓP + 1, while physical order still determines causal visibility.
- 4.2 Chunk boundaries: Each chunk boundary uses one shared position Mk for paired text and audio split markers, preventing timeline drift across segments.Text advances by character and audio by frame, so both streams jump to the same boundary coordinate.
- 4.3 Bounded context: CB0 retains one preceding chunk and limited lookahead, keeping transformer context bounded as passages grow.Its audio context contains only the previous chunk’s semantic tokens C0.
- 4.3 Bounded context: Inference chunks are capped at 350 characters and split at prioritized punctuation, then whitespace, with a hard split only as a last resort.The inference ceiling differs from the training chunk rule, which used a joint text-and-audio sequence budget.
- 4.4 Serving: vLLM adapters add a uniform prefill-coordinate offset so identity-indexed decoding preserves the intended relative positions.The adjustment accounts for skipped marker positions in the nonstandard coordinate layout.
5 Inference
Inference generates the semantic stream first, refines it through ordered acoustic stages, and reconstructs audio through overlapping windows in a causal latent space. This design supports streaming while maintaining bounded context and high throughput on one RTX 5090.
- Generation: CB0 generates coarse semantic codes until a split, end-of-speech marker, or configured generation limit, after which the stopping symbol is removed.The completed semantic row retains the corresponding boundary marker for later conditioning.
- Generation: Each acoustic stage generates exactly L tokens from completed lower rows, preserving the coarse-to-fine dependency order.The acoustic stages operate only after the relevant semantic and lower acoustic rows are complete.
- Generation: CB1–CB3 carry no autoregressive state across chunks, preventing cross-chunk acoustic-state accumulation and enabling concurrent refinement of available chunks.Within each chunk, however, acoustic stages remain strictly ordered.
- Reconstruction: Overlapping DualCodec reconstructions are re-encoded into VibeVoice latents, whose stable interiors are concatenated and decoded causally to avoid waveform seams.This enables streaming with DualCodec’s noncausal decoder without retraining it, at the cost of a second codec and withheld frames.
- Streaming: Streaming initially generates 40 semantic frames, withholds five unstable frames, and makes 35 frames, or 2.8 seconds, eligible for emission.Later semantic segments revise unstable boundaries and commit newly stable VibeVoice frames at two-second intervals.
- Serving performance: 200 ms to first encoded audio is achieved for one input on a warmed RTX 5090, excluding startup and model loading.The single-input end-to-end real-time factor is 0.08, while eight concurrent inputs reach an aggregate real-time factor of approximately 0.02.
6 Evaluation
TontaubeV1 is evaluated on 400 English audiobook passages using order-balanced Gemini pairwise judgments of prosody and correctness. It matches ElevenLabs on prosody and scores above the other comparators, while the protocol and scope impose important limitations.
- Protocol: The benchmark uses 400 English PG-19 passages, with each system synthesizing the same reference text under fixed sampling settings.Waveforms are independently normalized to −20 dBFS before judging.
- Protocol: Gemini judges prosody and word-level correctness independently across 800 order-balanced calls, while ignoring voice identity, timbre, and recording artifacts.Preference scores average values of 1, 1/2, or 0, and uncertainty resamples the 400 passages as paired clusters.
- Limitations: Comparisons with ElevenLabs, Gradium, and Cartesia may favor TontaubeV1 because it can inherit the cloned reference’s reading style while those systems use fixed provider voices.Voice identity and timbre were excluded from the rubric, but prosody is not fully separable from the cloned reference.
- Results: 50.1% prosody preference against ElevenLabs Flash v2.5 includes parity, while prosody scores are 82.1% against Fish Audio S2 Pro, 86.2% against the April 2026 Gradium API, and 82.3% against Cartesia Sonic 3.Correctness is statistically indistinguishable from parity against ElevenLabs, and its interval includes parity against Fish Audio.
- Results: 1.66% mean utterance-level WER is obtained on 1,088 English zero-shot Seed-TTS examples using Whisper large-v3 transcription.This result uses semantic sampling temperature 0.6.
- Validity: The exact Gemini 3.1 judging protocol is not claimed to be independently validated against human judgments.The paper also does not claim model judges are superior to human raters.
- Scope: The benchmark establishes English reading prosody and word-level correctness, not voice similarity, general sound quality, German or broader multilingual performance, long-form continuity, or streaming quality.These unmeasured properties remain outside the reported evaluation scope.
7 Limitations and Release
The system has technical, language, domain, safety, and licensing boundaries. These constraints require careful evaluation and responsible deployment.
- Autoregressive semantic generation can omit, repeat, or alter text and may terminate too early or too late.
- German phoneme realization is sometimes inaccurate, while the remaining supported languages have not been checked by native speakers.
- Audiobook-focused training may make audiobook generation more reliable than conversational or agentic generation.
- Voice cloning can enable impersonation, fraud, nonconsensual synthesis, and misleading media, so users remain responsible for consent.
- The Tontaube Community Model License 1.0 is not an open-source license and defines permitted uses and commercial requirements.
A Chunked generation example
Longer passages are divided into aligned chunks with shared text and audio boundaries. The current chunk receives previous context, limited lookahead, and audio rows that identify prior and current audio.
- Longer passages are divided into chunks whose matching boundaries appear in CB0’s text and audio rows.
- CB0 receives preceding text and audio context alongside the current chunk’s text and audio representation.
- The current chunk begins with a space, while only the final chunk carries a newline, encoding its position in the passage.
- The text row extends into the next chunk as lookahead, truncated at 50 characters before audio generation.
B Text verbalization
The optional verbalizer converts written English into spoken-form text for TontaubeV1. It expands pronunciation-sensitive forms but can alter wording, so exact-control callers may bypass it.
- The optional verbalizer maps written English to the spoken-form text X consumed by TontaubeV1.
- It expands numbers, dates, times, currencies, and symbols, standardizes abbreviations and initialisms, and otherwise preserves wording.
- The English-only verbalizer is independent of the four-stage graph and can normalize incorrectly or alter wording.
- Callers needing exact control can bypass the verbalizer and supply spoken-form text directly.
C LLM-as-a-Judge Instructions
The evaluation asks judges to compare two audios independently on prosody and word-level correctness. Judges must ignore sound quality, voice timbre, recording artifacts, and presentation order, then return raw JSON.
- Judges compare two audios independently on prosody and word-by-word correctness.
- Prosody covers rhythm, intonation, emphasis, pacing, and naturalness, while correctness measures accurate speech of the reference text.
- Judges must ignore hiss, compression, noise, clipping, timbre, and recording artifacts.
- Audio order must not influence the judgment.
- Responses must contain only a raw JSON object with prosody and correctness labels.