Source-linked AI summary
FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec
Zhihao Du, Shiliang Zhang, Kai Hu, Siqi Zheng
TL;DR
Neural codecs can provide discrete speech representations for speech-text modeling, but open-source toolkits have lacked a reproducible, integrable framework for developing and evaluating them in that context. FunCodec provides such a toolkit, including pretrained models and frequency-domain codecs; at matched token rates, it reports better quality than other open-source models for English and Mandarin speech.
Problem
Although neural codecs can provide discrete speech representations for generative speech-text modeling, open-source toolkits lack a reproducible, integrable framework for developing and evaluating them in that context.
Source: §1 · p. 1Method
Results
At the same token rate, FunCodec reports significantly better quality than other open-source models for English and Mandarin speech.
Source: §5.2 · p. 4Takeaways & Limitations
Abstract
from arXiv · showhide
This paper presents FunCodec, a fundamental neural speech codec toolkit, which is an extension of the open-source speech processing toolkit FunASR. FunCodec provides reproducible training recipes and inference scripts for the latest neural speech codec models, such as SoundStream and Encodec. Thanks to the unified design with FunASR, FunCodec can be easily integrated into downstream tasks, such as speech recognition. Along with FunCodec, pre-trained models are also provided, which can be used for academic or generalized purposes. Based on the toolkit, we further propose the frequency-domain codec models, FreqCodec, which can achieve comparable speech quality with much lower computation and parameter complexity. Experimental results show that, under the same compression ratio, FunCodec can achieve better reconstruction quality compared with other toolkits and released models. We also demonstrate that the pre-trained models are suitable for downstream tasks, including automatic speech recognition and personalized text-to-speech synthesis. This toolkit is publicly available at https://github.com/alibaba-damo-academy/FunCodec.
1. INTRODUCTION
Neural speech codecs have uses in speech-text modeling, but an open-source framework for reproducible development and evaluation in that context is lacking. FunCodec addresses this gap with a toolkit that also introduces frequency-domain models and semantic augmentation.
1. INTRODUCTION
- Open-source toolkits lack a reproducible, integrable framework for developing and evaluating neural speech codecs in speech-text modeling.The paper notes that codecs can provide discrete speech representations for generative models.Source: §1 · p. 1
- FunCodec provides recipes to train models from scratch or fine-tune pretrained ones, along with inference and evaluation scripts supporting GPU batch processing.The toolkit also releases pretrained academic and generalized models.Source: §1 · p. 1
- FreqCodec frequency-domain models aim for comparable performance with fewer parameters and lower computational complexity.Source: §1 · p. 1
- Semantic information improves speech quality at low bit rates, and the toolkit also evaluates codec models in downstream tasks.The paper lists ASR and personalized text-to-speech synthesis among its demonstrated applications.Source: §1 · p. 1
2. RELATED WORK
FunCodec is compared with other open-source codec toolkits, with its distinguishing features spanning pretrained models, training, and inference. The paper also describes frequency-domain and semantic-augmentation explorations.
2. RELATED WORK
- The comparison covers four GitHub toolkits, including AudioDec.Source: §2 · pp. 1–2
- FunCodec provides seven pretrained models for academic and generalized use, whereas the other toolkits offer a limited number.The models can serve as baselines or be applied directly to downstream tasks.Source: §2 · p. 2
- FunCodec offers single-stage training recipes, multiple discriminator options for speech quality, and distributed multi-GPU training.Source: §2 · p. 2
- Its inference processes all samples in a mini-batch simultaneously, and k-means codebook initialization is used to improve code utilization.Source: §2 · p. 2
3. FUNCODEC TOOLKIT
FunCodec combines model libraries with reproducible experiment recipes, supporting time- and frequency-domain codecs. Its architecture adds semantic augmentation to residual vector quantization and uses reconstruction, adversarial, and quantization-commit losses.
3. FUNCODEC TOOLKIT
- The toolkit pairs a PyTorch neural-model library with Kaldi-style Bash recipes for replicating experiments.The recipes are all-in-one scripts.Source: §3 · p. 2
3.1. Model architecture
- Time-domain models use an identity domain transformation, while frequency-domain FreqCodec models explore magnitude-angle and magnitude-phase representations.An encoder extracts acoustic representations, and a mirror-architecture decoder feeds a domain-inversion module that reconstructs waveforms.Source: §3.1 · p. 2
3.2. Semantic-augmented residual vector quantization
- Residual vector quantization produces discrete speech tokens; semantic augmentation can incorporate frame-aligned phoneme labels or Hubert embeddings.The toolkit also implements structured quantization dropout for a single model to operate across variable bitrates.Source: §3.2 · p. 2
3.3. Adversarial training objective with multiple discriminators
- Training combines reconstruction, adversarial, and RVQ commit losses, with frequency-domain reconstruction minimizing L1 and L2 distances on multiple Mel and magnitude spectra.The log-compressed power-spectrum loss is described as improving speech quality in middle and high frequencies; available adversarial discriminators include MSD, MPD, and MSTFTD.Source: §3.3 · p. 3
4. EXPERIMENTAL SETTINGS
Experiments use LibriTTS for academic models and a large bilingual in-house corpus for generalized models, with evaluation on multiple open-source corpora. Models are trained on clipped, RMS-normalized segments under specified GPU, batch-size, and update settings.
4.1. Experimental condition
- Academic models use 585 hours of LibriTTS, while generalized models train on 27.68 million bilingual utterances totaling approximately 25,000 hours.Generalized models are evaluated on Librispeech, aishell-1, aishell-2, Wenet, and Gigaspeech; all utterances are resampled to 16 kHz.Source: §4.1 · p. 3
- ViSQOL is the primary quality metric, and token ratio (TKR) measures tokens required per second at 16 kHz for fair comparisons.ViSQOL ranges from 1 to 5, with higher scores indicating better quality; at equal TKR, higher ViSQOL is preferred.Source: §4.1 · p. 3
4.2. Training details
- Training samples are randomly clipped to 3.2 seconds and RMS-normalized before encoding, with inverse normalization applied to reconstructed outputs for loss calculation.LibriTTS models use two V100 GPUs and batch size 32; generalized models use four A100 GPUs and batch size 128, with 300,000 codec-model updates.Source: §4.2 · p. 3
- The discriminator is updated only when its loss exceeds the codec model's loss, to prevent it from becoming too dominant.For FreqCodec, speech segments are transformed into spectrograms using a 512-sample window and 160-sample shift.Source: §4.2 · p. 3
5. EXPERIMENTAL RESULTS
FunCodec performs competitively across academic and generalized speech evaluations, with results covering frequency-domain models, semantic augmentation, and downstream tasks. The experiments report better quality at matched token rates, efficiency-quality trade-offs, and ASR performance that is sensitive to acoustic conditions.
5.1. Evaluation of academic models on LibriTTS corpus
- Reproduced SoundStream and Encodec reach state-of-the-art performance at different token rates; FunCodec-2x offers a favorable time–quantization-resolution balance compared with 4x.The log-compressed power-spectrum loss consistently improves speech quality, particularly at lower token rates; FunCodec-4x degrades at lower rates, so the authors adopt FunCodec-2x settings for generalized models.Source: §5.1 · p. 3
5.2. Evaluation of generalized models on multiple corpus
- At matched token rates, FunCodec achieves significantly better quality than other open-source models for English and Mandarin, though all models perform poorly on Wenet.Higher token rates consistently yield improved compressed quality; the authors attribute Wenet's poor results to complex recording environments and more non-speech noise.Source: §5.2 · p. 4, Figure 3 · p. 4
5.3. Comparison of frequency and time domain models
- FreqCodec achieves superior speech quality at higher token rates; magnitude with normalized phase is more suitable than magnitude with angle, and depthwise and grouped convolutions reduce complexity without compromising quality.Table 4 compares frequency- and time-domain models by ViSQOL score on LibriTTS; Mag denotes magnitude spectrogram and Cin the input channel number.Source: §5.3 · p. 4, Table 4 · p. 4
5.4. The impact of semantic augmentation
- Semantic tokens consistently improve quantized-speech quality, while residual combination is more suitable for speech codecs and offers an approach to reducing token rates.The experiments evaluate semantic information under low token rates.Source: §5.4 · p. 4, Table 5 · p. 4
5.5. Applying to down-stream tasks
- Codec tokens preserve much speech content and yield low ASR error, but are more sensitive to acoustic conditions than continuous fbank features; codebook embeddings are crucial for ASR.Table 6 compares inputs for ASR on Librispeech.Source: §5.5 · p. 4, Table 6 · p. 4
6. SUMMARY
FunCodec is presented as a reproducible, integrable open-source toolkit for neural speech codecs. The paper also evaluates frequency-domain models, semantic augmentation, and downstream ASR and TTS uses.
6. SUMMARY
- FunCodec is an open-source toolkit described as reproducible and integrable, with pre-trained academic and generalized models released alongside its codebase.The paper evaluates frequency-domain models and semantic augmentation based on the toolkit.Source: §6 · p. 4
- The paper evaluates FreqCodec frequency-domain models and augments the RVQ module with semantic information.Source: §6 · p. 4
- FunCodec models are validated in downstream tasks, including ASR and TTS.Source: §6 · p. 4