PaperlayerView exact revision on arXiv

Source-linked AI summary

RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

Chihiro Arata, Shohei Taniguchi, Gouki Minegishi, Kan Hatakeyama-Sato

arXiv:2610.29434v1cs.SDeess.AS

TL;DR

Threshold-based variable-rate codecs adapt token allocation to speech content but leave event counts dependent on the utterance, complicating prediction of sequence length. RubatoCodec combines content-adaptive placement with an exact utterance-level event budget and streaming decoding. The authors report competitive multilingual reconstruction, improved matched TTS metrics, and learned event budgets without an explicit generation-time rate constraint.

  • Problem

    Threshold-based allocation lets speech content determine event count, so sequence length varies across utterances and datasets and cannot be determined from audio duration alone.

    Source: §1 · pp. 1–2

  • Method

    RubatoCodec uses deterministic content-adaptive event placement with an exact utterance-level budget, represents events as discrete codes paired with durations, and decodes them causally.

    Source: §1 · p. 2, §9 · p. 33

  • Results

    The authors report competitive multilingual reconstruction, best intelligibility preservation among compared codecs on evaluated English and Japanese benchmarks at comparable bitrates, and improved matched TTS intelligibility and predicted speech quality with comparable speaker similarity.

    Source: §9 · p. 33

  • Takeaways & Limitations

    Discrete codes paired with durations let temporal resolution follow speech content while keeping sequence length predictable and support streaming text-to-speech generation.

    Source: §9 · p. 33

    TTS evidence covers one English audiobook corpus and controlled-comparison codecs; it does not isolate adaptive placement’s contribution or establish quality with multilingual codecs, broader data, or a larger generator.

    Source: §7.5 · p. 32

Abstract

from arXiv · show

Neural speech codecs provide discrete tokens for speech generation. Lower frame rates shorten token sequences, but each token must then carry more linguistic and acoustic information. Variable-frame-rate codecs address this trade-off by adapting token allocation to speech content, but threshold-based allocation lets content determine the token count, so the sequence length a speech generator must produce varies across utterances and datasets at a fixed setting. We present RubatoCodec, a variable-frame-rate speech codec built on X-Codec-2 with an exact utterance-level event budget and streaming decoding. Its Adaptive Firing mechanism uses changes in codec features and waveform spectra to divide speech into variable-duration events without a learned boundary predictor. Discrete codes and an explicit duration represent each event, allowing a speech generator to predict and decode one event at a time. Controlled experiments show that adaptive allocation outperforms fixed-rate coding in reconstruction fidelity and intelligibility at low event rates. A model trained on large-scale multilingual speech achieves competitive reconstruction quality across English, Japanese, and Mandarin, with the best intelligibility preservation among the compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates. In a matched text-to-speech comparison, the adaptive representation yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity. The speech generator also learns the codec's utterance-level event budget without an explicit generation-time rate constraint. These results demonstrate that budgeted adaptive speech representations support both high-quality reconstruction and streaming text-to-speech generation. Code and weights are available at https://github.com/third-intelligence/RubatoCodec and https://huggingface.co/third-intelligence/RubatoCodec.

1 Introduction

Lower-rate speech tokens shorten generation sequences but must represent longer acoustic intervals, while threshold-based variable-rate allocation leaves event counts dependent on speech content. RubatoCodec fixes the utterance-level event count while adapting event placement, and the paper reports reconstruction and TTS gains in its evaluations.

1 Introduction

  • Threshold-based allocation makes event counts vary across utterances and datasets, so audio duration alone cannot determine downstream sequence length or modeling cost.The introduction reports a range of 46.1–81.5 events/s for one codec under a single threshold across five datasets.Source: §1 · pp. 1–2
  • RubatoCodec sets K = round(RT) events for T latent frames, then uses Adaptive Firing to place variable-duration events from latent dissimilarity and waveform spectral flux.The default retention ratio R = 0.25 yields an average 12.5 events/s on a 50-Hz latent grid, without a learned boundary predictor.Source: §1 · p. 2
  • Each event carries discrete codes and an explicit duration, enabling a downstream model to predict and decode completed events incrementally.The codec builds on X-Codec-2 while adding adaptive aggregation and causal decoding.Source: §1 · p. 2
  • The authors report that adaptive allocation improves reconstruction over fixed-rate downsampling at low event rates and that matched TTS yields lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity.They also report competitive multilingual reconstruction and best intelligibility preservation among compared codecs on the evaluated English and Japanese benchmarks at comparable bitrates.Source: §1 · p. 2

2 Related Work

Prior work explores low-rate semantic codecs, variable-frame-rate allocation, and duration-aware or streaming generation. RubatoCodec combines a prescribed event count with content-dependent placement and explicit durations for causal decoding.

2.1 Neural speech codecs for language modeling

  • SoundStream, EnCodec, DAC, and BigCodec explore residual quantization or high-capacity codebooks, while semantic supervision in SpeechTokenizer, Mimi, and X-Codec-2 aims to retain linguistic content.RubatoCodec builds on X-Codec-2’s acoustic/semantic architecture and uses serial residual quantization.Source: §2.1 · pp. 2–3

2.2 Variable-frame-rate speech coding

  • Variable-rate codecs adapt temporal resolution through hierarchy selection, merging, or boundary selection, while methods differ in how they control event counts and encode timing.RubatoCodec uses deterministic statistics for boundaries and an aggregator that combines onset features, enclosed-frame means, and duration.Source: §2.2 · p. 3

2.3 Duration-aware and streaming speech generation

  • Duration-aware and streaming systems model timing alongside content, and RubatoCodec supplies explicit event durations to a causal waveform decoder for incremental synthesis.A downstream model predicts code–duration pairs that restore the decoding grid as events become available.Source: §2.3 · pp. 3–4

3 RubatoCodec Architecture

RubatoCodec modifies X-Codec-2’s temporal bottleneck with content-adaptive event placement, duration-aware representations, and causal decoding. Its architecture retains acoustic and semantic encoding and spectral synthesis while supporting a fixed event budget and incremental output.

3.1 Representation and signal path

  • The codec maps audio to a nominal 50-Hz feature grid and represents each nonempty event with quantizer indices and a duration in 20-ms frames.Its X-Codec-2-based architecture retains acoustic/semantic encoding and Transformer-based spectral synthesis, while adding Adaptive Firing, interval aggregation, hybrid residual quantization, and duration-aware grid reconstruction.Source: §3.1 · p. 4, §3 · p. 4, §3.2 · p. 5

3.3.1 Firing signal

  • Adaptive Firing combines multi-scale latent dissimilarity and log-mel spectral flux to determine event boundaries; the firing signal selects boundaries rather than weighting pooled features.Sequence-wise mean normalization makes the combination depend on the components’ relative temporal distributions, and waveform flux provides an allocation signal before acoustic features become meaningful.Source: §3.3 · p. 5, §3.3.1 · pp. 6–7

3.3.2 Budget-constrained event allocation

  • For retention ratio R, the integer event budget is set over T encoder frames; cumulative bounded increments and position correction produce exactly K distinct event starts.Allocation is deterministic and non-differentiable, with no rate loss; it enforces total count without imposing a maximum event duration, and preserves K nonempty events in specified edge cases.Source: §3.3.2 · pp. 7–8

3.3.3 Interval aggregation; 3.4.1 Residual quantization

  • Each event’s quantizer input combines its onset feature with a learned correction based on the event’s frame-feature mean and duration; residual quantization adds code indices in successive layers.Using the first D quantizer layers changes the number of code indices per event while retaining the same boundaries and durations.Source: §3.3.3 · p. 8, §3.4.1 · p. 9

3.4.2 Semantic distillation

  • A training-only semantic head predicts frozen w2v-BERT 2.0 features from the first-layer reconstruction, with masked targets at event starts.The semantic objective encourages the first layer to retain linguistic information; residual layers encode additional waveform detail.Source: §3.4.2 · p. 9

3.5 Causal grid decoding

  • The decoder places each dequantized event vector at its start and uses a learned empty vector between starts; causal attention and convolution restrict synthesis to current and preceding events.Streaming inference processes incoming events incrementally, reuses decoder states, and periodically rebuilds cached states from recent events with a short audio crossfade.Source: §3.5 · pp. 9–10

3.6 Model size and bitrate

  • The codec has approximately 246.1M parameters, while the complete model including the frozen semantic encoder has approximately 826M; token-to-waveform synthesis uses the decoding path and quantizer.At 12.5 events/s, quantizer depths 1, 4, 8, and 16 correspond to content bitrates of 0.15, 0.60, 1.20, and 2.40 kbit/s, respectively.Source: §3.6 · p. 10

4 Training Data and Optimization

The section describes staged codec objectives and training setups for adaptive-versus-fixed-rate comparisons, a placement-only comparison, and multilingual training.

4.1 Objectives and phases

  • Training progresses from mel, semantic, and quantization losses to adversarial and feature-matching losses, then adds perceptual loss in the final phase.The phases occupy the first 10%, next 75%, and final 15% of updates, respectively.Source: §4.1 · pp. 10–11

4.2 Training for architecture comparisons

  • Adaptive and fixed-rate models are compared on the same approximately 5,000-hour English subset at 6.25, 12.5, and 25 events/s, with shared objectives and separately trained rate configurations.The adaptive models use retention ratios of 0.125, 0.25, and 0.5, respectively.Source: §4 · p. 10, §4.2 · p. 11
  • The placement-only comparison trains adaptive and uniform-stride models on the full English 44-kHz HiFiTTS-2 partition for one million steps at 12.5 Hz.Both use the same 50-Hz encoder and decoder grids, interval aggregator, and hybrid residual quantizer; the uniform control forms four-frame events.Source: §4 · p. 10, §4.2 · p. 11

4.3 Multilingual model training

  • The multilingual model uses approximately 375,000 hours of speech data, with English, Japanese, and Chinese comprising approximately 44%, 27%, and 13% of the mixture.It is trained for 1.5 million steps at retention ratio 0.25, using aligned six-second crops resampled to 16 kHz for encoding and 44.1 kHz for reconstruction targets.Source: §4.3 · p. 12

5 Evaluation

The evaluation examines RubatoCodec across five datasets and 10,563 utterances, measuring intelligibility, speech quality, and speaker preservation in controlled and multilingual comparisons. It also assesses adaptive event allocation under the fixed utterance-level budget and reports decoding, streaming, and encoding performance.

5.2.1 Adaptive versus fixed-rate architectures

  • At 12.5 and 6.25 Hz, adaptive coding improves every reported metric except SIM; at 25 Hz, fixed coding has slightly lower dWER at both depths.At 12.5 Hz, depth-16 dWER falls from 0.72% to 0.62%; at 6.25 Hz it falls from 1.24% to 1.08%. The 12.5-Hz gains combine interval aggregation, adaptive placement, and decoding on a 50-Hz grid.Source: §5.2.1 · p. 15, Table 6 · p. 15, Table 7 · p. 16

5.2.2 Uniform-stride versus adaptive event placement

  • With aggregation and decoding held fixed, adaptive event placement yields higher SIM at every depth and lower dWER at all but depth 8, while MOS predictors show no consistent ordering.With one codebook, dWER falls from 17.00% to 11.64%; at depths 8 and 16, the values are close.Source: §5.2.2 · p. 18, Table 8 · p. 18

5.2.3 Event rate and quantizer depth at similar bitrates

  • Near 1.2 kbit/s, 12.5 Hz at depth 8 nearly matches 25 Hz at depth 4 in PESQ-WB and has slightly lower dWER; near 2.4 kbit/s, 25 Hz at depth 8 leads in PESQ-WB.At 12.5 Hz, sequence length is half that at 25 Hz near 1.2 kbit/s; higher event rates improve fidelity and speaker similarity, while substantial rate reductions sharply degrade quality.Source: §5.2.3 · p. 19, Table 9 · p. 19

5.3.1 English

  • On LibriSpeech and VCTK, RubatoCodec has the lowest dWER and highest UTMOSv2 in both bitrate bands, while stronger comparators retain higher fidelity or speaker similarity.On LibriSpeech at depth 16, dWER is 0.50%; on VCTK it is also 0.50%.Source: §5.3.1 · pp. 19–20, Table 10 · p. 20, Table 11 · p. 21

5.3.2 Japanese and Mandarin Chinese

  • On JVS nonpara30 and JVNV, RubatoCodec has the lowest dCER in both groups; on Mandarin it is competitive, but leading codecs outperform it in both bands.On JVNV, dCER is 1.92% versus 5.44% at depth 8 and 1.54% versus 3.01% at depth 16; on Mandarin at depth 16 it is 0.50%, behind MOSS at 0.38% and Qwen at 0.45%.Source: §5.3.2 · p. 21, Table 12 · p. 22, Table 13 · p. 23

5.4.1 Event-rate control across datasets

  • RubatoCodec and CodecSlime stay near prescribed event rates, whereas threshold-based DTM-Codec and FlexiCodec rates vary across datasets; DyCAST character-aligned rates also vary by dataset and language.DTM-Codec’s nominal 80-Hz setting ranges from 46.1 to 81.5 events/s, and FlexiCodec at a fixed threshold ranges from 7.0 to 8.5 events/s. DyCAST character-aligned rates range from 11.0–14.2 Hz on English datasets to 3.7–4.4 Hz on the other datasets.Source: §5.4.1 · p. 24, Table 15 · p. 25

5.4.2 Adaptive Firing on real speech

  • Across five datasets, event durations have an 80-ms median; 99.98% of events longer than 300 ms are predominantly low-energy.JVS and JVNV have 60-ms medians but higher 95th percentiles than the English and Mandarin datasets. The low-energy classification uses an amplitude-based near-silence proxy, not annotated speech activity or phonetic categories.Source: §5.4.2 · p. 25, Figure 9 · p. 25, Table 16 · p. 26
  • Longer-duration events were associated with low-energy regions while retaining the prescribed utterance-level event budget.The remaining events had 25.2% low-energy frames.Source: §5.4.2 · p. 26

5.5 Computational efficiency

  • 446.8× and 427.4× real-time decoding at depths 8 and 16 show that quantizer depth has little effect on RubatoCodec’s 44.1-kHz decoding throughput.Full-utterance throughput was measured on one NVIDIA H200 using batch size one.Source: §5.5 · p. 27, Table 17 · p. 27
  • Approximately 71× real-time encoding included the frozen semantic feature extractor, while depth-8 RubatoCodec took approximately 2.5% longer than X-Codec-2 to encode the same audio.The comparison does not separate Adaptive Firing from quantization differences.Source: §5.5 · p. 27, Table 17 · p. 27
  • 11.07 ms per event was the average streaming decode time, versus approximately 80 ms of audio per event at 12.5 events/s, supporting real-time streaming on the tested hardware.The 250-event test used codes from synthetic audio and included history recomputation and crossfading.Source: §5.5 · p. 27

6 Duration-Aware Text-to-Speech Modeling

RubatoCodec events are generated autoregressively as code–duration pairs and decoded incrementally. In the matched English TTS comparison, the adaptive codec improves WER and predicted quality; generated event counts also closely follow the codec’s budget rule without a generation-time rate constraint.

6.1 Generative interface

  • The temporal Transformer predicts events from the conditioning prefix and completed events, while the local Transformer predicts continuation or stop, codebook tokens, and duration.Completed code–duration pairs are passed to the causal codec decoder for incremental synthesis.Source: §6.1 · p. 27, Figure 11 · p. 28

6.2 TTS evaluation

  • The matched TTS evaluation uses zero-shot voice cloning on 1,088 English seed-tts-eval utterances, with shared target texts and reference recordings.WER, speaker similarity, UTMOS, and UTMOSv2 are averaged over utterances.Source: §6.2 · p. 29
  • The adaptive codec achieves lower WER and higher UTMOS and UTMOSv2, with comparable speaker similarity, than the fixed-rate codec.The matched systems use eight codebooks at a nominal event rate of 12.5 Hz.Source: §6.2 · p. 29, Table 18 · p. 29

6.3 Generated event timing

  • For 88.3% of utterances, generated event counts equal round(0.25T), and 99.9% fall within one event, without an explicit generation-time rate constraint.Generated rates average 12.461 events/s; individual utterance rates range from 11.897 to 12.849 events/s.Source: §6.3 · p. 29, Figure 12 · p. 30
  • Across 61,965 generated events, mean duration is 80.24 ms and the 5th–95th percentile interval is 40–140 ms; 362 events exceed 300 ms.Of those 362 long events, 361 meet the predominantly low-energy criterion; informal listening also found an unnatural within-phrase pause associated with a long event.Source: §6.3 · p. 29, Figure 13 · p. 30

7 Discussion

An exact event budget permits content-dependent placement, but its trade-offs differ from threshold allocation and vary with temporal resolution. The discussion also identifies limits in multilingual and TTS evaluation scope, including the need for human assessment of pause naturalness.

7.1 Adaptive allocation under an explicit budget

  • An exact budget preserves predictable sequence lengths while allowing content-dependent placement, but equally long utterances receive the same budget regardless of acoustic complexity.Long events predominantly cover low-energy regions, leaving more events for the remaining speech; threshold methods can instead spend more events on difficult utterances.Source: §7.1 · p. 31

7.2 Temporal resolution and quantizer depth for speech generation

  • At the evaluated settings, additional residual layers do not fully compensate for coarser segmentation, and higher inference-time rates are better tolerated than aggressive rate reductions.The reconstruction comparisons motivate 12.5 Hz as a practical compromise, not an established optimum; shorter sequences reduce downstream modeling cost but not waveform-decoder computation.Source: §7.2 · p. 31

7.3 From encoded durations to generated timing

  • Generated event counts match the budget rule for 88.3% of utterances and fall within one event for 99.9%, without a generation-time rate constraint.Long generated events predominantly correspond to low-energy regions, but informal listening found an unnatural within-phrase pause associated with a long event.Source: §7.3 · pp. 31–32

7.4 Reconstruction quality and multilingual performance

  • RubatoCodec shows cross-language reconstruction with a shared model and allocation rule, but higher predicted quality does not imply greater fidelity to the source.The evaluated languages are all in the training mixture, and the 44.1-kHz output rate does not establish full-band source recovery from 16-kHz input.Source: §7.4 · p. 32

7.5 Scope and limitations

  • The placement comparison uses one seed and English read speech, leaving robustness across runs, languages, and recording conditions untested.The reported architecture comparisons use one trained model per configuration, and firing statistics do not establish phonetic alignment or optimal allocation.Source: §7.5 · p. 32
  • Because all evaluated languages occur in the training mixture, the multilingual results do not establish unseen-language transfer or language robustness attributable solely to Adaptive Firing.Dataset differences also confound language, speakers, and recording conditions.Source: §7.5 · p. 32
  • TTS evidence covers one English audiobook corpus and the controlled-comparison codecs; human evaluation remains necessary for preferences and pause naturalness.The comparison does not isolate the TTS contribution of adaptive placement or establish quality with the multilingual codec, broader data, or a larger generator.Source: §7.5 · p. 32

8 Future Work

Future work targets codec fidelity at low event rates, broader testing of TTS and Adaptive Firing, and online encoding. These directions include balancing buffering latency against placement quality and rate control.

8 Future Work

  • For the codec, future work should test whether improved acoustic representations and reconstruction objectives can close PESQ/STOI gaps at low event rates, and when finer temporal resolution is necessary.Training with multiple retention ratios could broaden the observed inference-time tolerance; online encoding would require a causal acoustic–semantic path and a windowed or running budget.Source: §8 · p. 33
  • Larger-scale TTS with the multilingual codec, broader data, and larger backbones could test whether reconstruction improvements carry over.Further tests include explicit durations for utterance-length and pause control, and applying Adaptive Firing to other codec backbones, music, or environmental sound.Source: §8 · p. 33

9 Conclusion

RubatoCodec combines content-adaptive event placement, an exact utterance-level event budget, and streaming waveform decoding. Evaluations support its use for reconstruction and speech generation, while signal fidelity remains an area for improvement.

9 Conclusion

  • RubatoCodec pairs discrete codes with durations, allowing temporal resolution to follow speech content while keeping sequence length predictable.The codec combines an exact utterance-level event budget with streaming waveform decoding.Source: §9 · p. 33
  • Among compared codecs at comparable bitrates, RubatoCodec has the best intelligibility preservation on evaluated English and Japanese benchmarks, while signal fidelity remains an area for improvement.Matched text-to-speech experiments show lower WER and higher UTMOS and UTMOSv2 with comparable speaker similarity; the generator learns the utterance-level event budget without an explicit generation-time rate constraint.Source: §9 · p. 33
  • Controlled experiments support adaptive allocation for reconstruction at low event rates, and the results support budgeted adaptive tokenization for reconstruction and speech generation.The authors describe combining content-dependent timing with predictable sequence length as a basis for compact speech modeling and streaming synthesis.Source: §9 · p. 33

Generative AI Use Disclosure

The appendix discloses writing-assistance use and reports exploratory comparisons of firing signals, decoder causality, event representations, and firing rules. Across these ablations, the selected design balances reconstruction quality with architectural simplicity, while causal decoding and cumulative allocation have supported trade-offs or advantages in the tested settings.

Generative AI Use Disclosure

  • The authors report that generative AI tools assisted with English expression, clarity, and readability, while the authors developed the ideas, methods, and conclusions and reviewed the text.Source: Generative AI Use Disclosure · p. 33

A.1 Firing-signal selection

  • Multi-scale latent dissimilarity combined with spectral flux achieves the highest PESQ-WB, with STOI and speaker similarity comparable to flux alone, although dWER is slightly higher.The authors adopt this combined signal as a balance between architectural simplicity and reconstruction quality in the exploratory comparison.Source: §A.1 · p. 38

A.2 Effect of causal decoding

  • Causal decoding reduces PESQ-WB by 0.099 and 0.095, STOI by 0.006 and 0.004, and SIM by 0.035 and 0.029 at depths 8 and 16, respectively.dWER is 0.80% versus 0.76% at depth 8 and 0.62% versus 0.60% at depth 16 for causal versus non-causal decoding; the results support causal synthesis with modest reference-based quality losses.Source: Table A2 · p. 39, §A.2 · p. 39

A.3 Sparse versus repeated event representations

  • Sparse event-representation composition improves PESQ-WB and STOI and lowers dWER at both depths, while repeated representations yield slightly higher SIM and UTMOSv2.The authors interpret the results as supporting sparse placement for signal fidelity.Source: Table A3 · p. 39, §A.3 · p. 39

A.4 Firing decision rules

  • Cumulative allocation outperforms level thresholding on PESQ-WB, STOI, speaker similarity, UTMOSv2, and DNSMOS, and yields lower dWER at both tested depths.The comparison evaluates complete firing rules, which differ in event-count enforcement and training gradients; it does not isolate which difference explains the improvement.Source: Table A4 · p. 40, §A.4 · p. 40

A.5 Inference-time event-rate variation

Inference-time rate changes reveal asymmetric tolerance around the training rate, while the duration-order comparison favors duration-last prediction under its evaluated metrics. The appendix also points to separate training schedules for the codec configurations.

A.5 Inference-time event-rate variation

  • At depth 8, reducing the rate from 12.5 to 10 Hz lowers PESQ-WB from 2.684 to 2.421 and raises dWER from 0.75% to 1.11%; at 5 Hz, dWER reaches 45.66%.Increasing rate above 12.5 Hz improves PESQ-WB, STOI, and SIM with diminishing gains; at depth 8, dWER reaches 0.56% at 20 Hz, while bitrate rises from 1.23 to 1.96 kbit/s between 12.5 and 20 Hz. The authors report partial tolerance, particularly for rate increases.Source: §A.5 · pp. 40–41, Table A5 · p. 41

B Duration Prediction Order

  • Duration-last prediction has lower duration cross-entropy and TTS WER in the small-scale comparison, but its cross-entropy is not directly comparable across prediction orders.Duration-last conditions on all current-event codes, which may retain duration information from interval aggregation.Source: B Duration Prediction Order · p. 41, Table B1 · p. 42

C.1 Codec training

  • The appendix identifies separate optimizer learning-rate schedules for each training configuration and notes that discriminator updates begin in the adversarial phase.Table C1 provides detailed settings for the architecture and placement comparisons and the multilingual model.Source: §C.1 · p. 42

C.2 TTS training and sampling

The appendix records shared TTS training and sampling settings, then lists public sources for study components and identifies X-Codec-2 as RubatoCodec’s architecture backbone.

C.2 TTS training and sampling

  • The TTS comparison initializes its temporal backbone and text embeddings from pretrained Qwen3-0.6B weights, while codec-specific embeddings, the local Transformer, and prediction heads are newly initialized.Losses apply only to target positions, and head losses are averaged equally; evaluation sampling uses specified temperature, top-p, top-k, and repetition-penalty settings.Source: §C.2 · p. 42, Table C2 · p. 44

D Code, Models, and Data

  • The appendix lists public sources for codec implementations, pretrained components, evaluation tools, and datasets used in the study.Experimental configurations and data selection are described in Sections 4–6 and Appendix C.Source: D Code, Models, and Data · p. 44

D.1 Codec implementations and checkpoints

D.2 Datasets

The dataset note says the multilingual mixture includes internal data, while the evaluation-tools subsection identifies the speaker-similarity checkpoint and UTMOS model.

D.2 Datasets

  • The multilingual training mixture includes internal data, as described in Section 4.3.Source: §D.2 · p. 45

D.3 Pretrained components and evaluation tools

  • Speaker similarity uses a WavLM-Large-based speaker-verification setup with the UniSpeech ECAPA-TDNN head.Source: §D.3 · p. 46
  • UTMOS is measured with the utmos22_strong model from SpeechMOS v1.2.0.Source: §D.3 · p. 46
Loading 2610.29434v1…