ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech

• Neural codecs use encoder-decoder architectures with quantizers to produce discrete codes, extending their original compression role to speech and audio tasks.Source: §1 · p. 1
• Variations in training data, environments, and evaluation settings make controlled codec comparisons difficult and can affect downstream experimental findings.Source: §1 · p. 1
• ESPnet-Codec provides five neural codec algorithms with all-in-one recipes covering data preparation, training, inference, and evaluation across audio, music, and speech.Source: §1 · p. 1
• VERSA evaluates existing codec algorithms in a controlled environment, while experiments show no single model is best across all metrics.Source: §1 · p. 1
• ESPnet-Codec integrates pre-trained codecs into six downstream tasks and finds correlations between codec performance and downstream effectiveness, including feasible use in understanding tasks.Source: §1 · p. 1

• Existing toolkits often lack complete data, training, inference, evaluation, and pre-trained-model pipelines, complicating comparisons across models.Source: §2 · pp. 1–2
• Evaluation toolkits commonly support only a few metrics, which can introduce bias, whereas broader metric coverage enables more comprehensive codec analysis.Source: §2 · p. 2
• Current downstream integrations focus mainly on generation, leaving diverse codec-token uses and understanding tasks insufficiently represented.Source: §2 · p. 2
• ESPnet-Codec reimplements codec algorithms within the ESPnet ecosystem, provides transparent comparisons through ESPnet-style recipes, and evaluates them with VERSA.Source: §2 · p. 2
• The toolkit supports ASR, TTS, speaker recognition, speech separation and enhancement, singing voice synthesis, and self-supervised pre-training through a discretization interface.Source: §2 · p. 2

3.1. General Neural Codec Framework

• ESPnet-Codec defines a serial codec pipeline that encodes sampled audio, quantizes hidden states into multi-codebook discrete codes, and decodes them into reconstructed signals.It supports waveform or STFT representations, primarily RVQ quantization, convolutional encoder-decoder variants, and GAN-based reconstruction, adversarial, and quantization objectives.Source: §3.1 · p. 2

3.2. Example Neural Codec Models

• The framework implements representative neural codecs including SoundStream, Encodec, DAC, FunCodec, and HiFi-Codec, preserving their differing quantizers, discriminators, and training designs.Examples include GRVQ in HiFi-Codec, factorized and normalized codes in DAC, and frequency-domain training in FunCodec.Source: §3.2 · p. 3

3.3. Speech and Audio Quality Evaluation with VERSA

• VERSA unifies codec evaluation across intrusive, non-intrusive, and perceptual metrics, addressing risks from limited metric sets and inconsistent evaluation conditions.Its supported measures include signal-quality, intelligibility, speaker-similarity, error-rate, and learned perceptual metrics.Source: §3.3 · p. 3, Table 2 · p. 3

3.4. Downstream Application

• ESPnet-Codec integrates codec tokens into ASR, NAR and AR TTS, speaker recognition, speech separation and enhancement, singing voice synthesis, and SSL workflows.The implementations replace or supplement conventional acoustic inputs with discrete codec representations across these downstream tasks.Source: §3.4 · pp. 3–4

4.1. Codec Dataset

• LibriTTS provides 460 hours for training, while AMUSE combines open-source speech, audio, and music datasets and evaluates both 16kHz and 44.1kHz settings.AMUSE development and test sets are sampled from its constituent datasets, with additional singing-voice test data.Source: §4.1 · p. 4

4.2. Codec Experimental Setup

• The base setting uses one V100 GPU and 1-second chunks, whereas the large-scale setting uses four Ampere GPUs and 2-second chunks.Both settings use 600k training steps, while batch sizes are 8 and 128, respectively.Source: §4.2 · p. 4
• All codecs use 32-level residual-vector-quantization codebooks with 1,024 codes and multi-band training at randomly controlled bitrates from 2 to 16 kbps.Model loss weights are tuned on development sets, and configurations mostly follow original papers or released implementations.Source: §4.2 · p. 4
• The AMUSE corpus adds high-quality TTS-focused speech data because its original speech collection contains varied quality from upsampling low-rate signals.This adjustment targets balance in the large-scale training data.Source: §4.2 · p. 4
• Two implementation changes are reported: Encodec omits its loss balancer, and DAC omits code-space factorization because low-dimensional hidden states were difficult to converge.These choices were based on development-set effectiveness and empirical convergence behavior.Source: §4.2 · p. 4
• The experiments reproduce DAC, Encodec, FunCodec, and SpeechTokenizer on LibriTTS, while evaluation standardizes resampling and task-specific metric procedures.CER uses Whisper-large-V3, and speaker similarity uses cosine similarity between RawNet3 embeddings.Source: §4.2 · p. 5

4.3. Codec Results and Discussion

• Encodec leads most metrics among ESPnet-Codec models on LibriTTS, while SoundStream has stronger F0 tracking and DAC has better 24kHz PESQ.The reproduced models otherwise achieve performance comparable to existing codec implementations.Source: §4.3 · p. 5
• DAC outperforms several metrics but has much worse Whisper WER than FunCodec, demonstrating metric-dependent trade-offs among codec models.This contrast motivates evaluating codecs with a broad metric suite rather than a single measure.Source: §4.3 · p. 5
• Large-scale AMUSE data substantially changes codec rankings across speech, audio, and music, while no single model achieves the best scores on every metric.Encodec performs well on LibriTTS but generalizes less effectively to AMUSE speech.Source: §4.3 · p. 5, Table 7 · p. 5

4.4. CodecSUPERB Evaluation

• Encodec trained on base LibriTTS performs best across most speech-related CodecSUPERB metrics, whereas SoundStream trained on AMUSE performs better on audio-related metrics.The benchmark reports WER, equal error rate, and accuracy for downstream applications.Source: §4.4 · p. 5
• Scaling training data improves SoundStream and DAC but does not necessarily improve Encodec, especially on speech-related metrics.The results suggest that larger datasets may require additional strategies because codec architectures are sensitive to diverse scenarios.Source: §4.4 · p. 5

4.5. Downstream Integration: ASR

• Encodec trained on LibriTTS achieves the best discrete-ASR performance, while SoundStream is strongest among codecs trained on AMUSE.Experiments use LibriSpeech 960 and report WER on test-clean and test-other.Source: §4.5 · pp. 5–6
• The ASR results indicate that codec data scaling is not consistently predictive of downstream performance and that audio-quality metrics correlate only weakly with it.Discrete codes are converted to embeddings by looking up codec codebooks before entering the ASR system.Source: §4.5 · p. 6

4.6. Downstream Integration: TTS

• LibriTTS-based codecs consistently outperform AMUSE counterparts in both NAR-TTS and AR-TTS.The authors attribute this mainly to consistency between codec and downstream TTS training datasets.Source: §4.6 · p. 6
• SoundStream achieves the best intelligibility in NAR-TTS, whereas Encodec achieves the best UTMOS.Source: §4.6 · p. 6

4.7. Downstream Integration: SSE

• DAC trained on AMUSE achieves the best PSEQ and STOI, while Encodec trained on LibriTTS achieves the highest DNSMOS in speech separation.Scaling from LibriTTS to AMUSE improves DAC performance but not necessarily SoundStream or Encodec performance.Source: §4.7 · p. 6

4.8. Downstream Integration: SVS

• SoundStream is the better overall singing voice synthesis choice because it outperforms Encodec on SACC and SingMOS despite a worse MCD score.Source: §4.8 · p. 6
• Singing voice synthesis depends on codec-token prediction difficulty as well as the codec’s upper-bound resynthesis quality.Source: §4.8 · p. 6

4.9. Downstream Integration: SPK

• AMUSE-trained codec tokens consistently outperform LibriTTS-trained tokens for speaker verification, confirming that codec tokens encapsulate speaker information.Significant overfitting was also observed, motivating future research in codec-based speaker verification.Source: §4.9 · p. 6

4.10. Downstream Integration: SSL

• Codec-input SSL pre-training requires 200 GPU hours versus 280 hours for waveform inputs in an equivalent data setting.Source: §4.10 · p. 6
• Quantized codec inputs can cause unstable SSL pre-training, with certain codecs failing even after hyperparameter tuning.Source: §4.10 · p. 6

5. CONCLUSION

• ESPnet-Codec focuses on neural codec training and evaluation across diverse scenarios, complemented by VERSA for unified comprehensive speech evaluation.Source: §5 · p. 6
• Integration into six downstream tasks demonstrates the framework’s applicability while enabling extensive probing and analysis of neural codecs.Source: §5 · p. 6