Source-linked AI summary
ZipCodec: Ultra-Low-Frame-Rate Streaming Speech Coding
Luca Della Libera, Cem Subakan, Mirco Ravanelli
TL;DR
ZipCodec tackles the difficulty of preserving rich speech information at very low frame rates. It combines large-scale WavLM distillation with a redesigned streaming architecture and spherical quantization, achieving strong reconstruction and downstream performance at 6.25 Hz and 0.80 kbps. Despite its size, it also supports real-time single-stream CPU inference.
Problem
Simultaneously achieving low bitrate, rich acoustic and semantic representations, high reconstruction quality, and streamability remains challenging as lower frame rates increase each token’s information burden.
Method
ZipCodec combines large-scale WavLM layer-6 distillation with a redesigned transformer-based streaming architecture, scalar spherical quantization, and a latency-aware decoder.
Results
ZipCodec consistently improves reconstruction and downstream-task performance over streaming baselines at matched bitrates while operating at 6.25 Hz.
Takeaways & Limitations
ZipCodec provides compact 6.25 Hz representations that retain rich speech information and shorten sequences by 8 times relative to 50 Hz streaming representations.
Abstract
from arXiv · showhide
Neural audio codecs are a fundamental component of modern speech generation systems. While recent codecs achieve increasingly low bitrates, reducing frame rate remains challenging, as each token must preserve more information while maintaining reconstruction quality. We present ZipCodec, a streaming neural speech codec operating at 6.25 Hz and 0.80 kbps with a theoretical latency of 160 ms. Our approach combines large-scale WavLM distillation with a redesigned transformer-based architecture, a scalar spherical quantizer, and a latency-aware streaming decoder. Experiments show that ZipCodec substantially outperforms existing streaming codecs at comparable bitrates in both reconstruction and downstream tasks, while operating at a significantly lower frame rate. Despite its 842M parameters, ZipCodec achieves real-time single-stream inference on a consumer-grade CPU. Demo samples, code and checkpoints are available at https://lucadellalib.github.io/zipcodec-web/.
1. INTRODUCTION
ZipCodec addresses the challenge of combining low bitrate, rich speech representations, reconstruction quality, and streamability at an exceptionally low frame rate. It operates at 6.25 Hz and 0.80 kbps while improving reconstruction and representation quality over streaming codecs.
- 1. INTRODUCTION: Reducing frame rate shortens token sequences and lowers computational cost, but forces each token to encode more linguistic, speaker, prosodic, and acoustic information.
- 1. INTRODUCTION: 6.25 Hz pushes streaming speech coding below the 12.5 Hz floor previously demonstrated for streaming codecs.
- 1. INTRODUCTION: 6.25 Hz and 0.80 kbps enable ZipCodec to reduce frame rate while retaining strong reconstruction and representation quality in a streamable codec.Its theoretical latency is 160 ms, and the codec remains compatible with responsive streaming speech-to-speech systems.
- 1. INTRODUCTION: ZipCodec combines WavLM layer-6 distillation, an optimized transformer architecture, scalar spherical quantization, and a latency-aware streaming decoder.Distillation is scaled to approximately 94,000 hours of English speech with WavLM-style augmentation.
- 1. INTRODUCTION: Experiments evaluate reconstruction and downstream tasks, showing substantial improvements over streaming codecs at matched bitrates despite the lower frame rate.
2. ZIPCODEC
ZipCodec uses a causal encoder–compressor–quantizer–decompressor–decoder pipeline to transform speech into compact 6.25 Hz representations and resynthesize it causally. Its architecture combines temporal patching, transformer processing, factorized spherical quantization, and streaming waveform decoding.
- 2. ZIPCODEC: ZipCodec contains five causal components: encoder, compressor, quantizer, decompressor, and decoder, with a non-causal teacher used for feature distillation.
- 2.1. Architecture: Temporal patching groups 16 consecutive 100 Hz log-mel frames, reducing the representation rate to 6.25 Hz over 160 ms intervals.
- 2.1. Architecture: The decompressor expands each 6.25 Hz representation into eight 50 Hz WavLM layer-6 representations before waveform decoding.Each discrete representation spans 160 ms and maps to eight continuous representations spaced 20 ms apart.
- 2.1. Architecture: A streaming Vocos decoder maintains overlap-add state and jointly processes the eight WavLM representations available within each 160 ms interval.
- 2.2. Training: ZipCodec jointly trains the encoder, compressor, quantizer, and decompressor through WavLM distillation while separately training the waveform decoder.Noise and overlapping-speech augmentation reproduce aspects of the WavLM pretraining distribution.
3. EXPERIMENTAL SETUP
The experiments train ZipCodec on large speech corpora and evaluate it against streaming codecs at comparable low bitrates. The setup also measures streaming resource use across CPU and GPU configurations.
- 3. EXPERIMENTAL SETUP: Training uses 40.96-second segments from LibriLight, VoxPopuli, and GigaSpeech, with short utterances discarded and longer or shorter utterances cropped or repeated.
- 3. EXPERIMENTAL SETUP: The compressor and decompressor use six ErfFormer blocks with 2048-dimensional model states and bounded 256-frame streaming caches.The cache provides 40.96 seconds of context at 6.25 Hz, matching the training context length.
- 3. EXPERIMENTAL SETUP: The waveform decoder is trained on LibriTTS-100 and uses 20 ConvNeXt blocks, explicit convolutional and overlap-add state, and streaming inverse-STFT synthesis.
4. RESULTS
ZipCodec delivers strong reconstruction and downstream performance at 6.25 Hz and 0.80 kbps, while remaining streamable and real-time efficient despite its large model size.
- 4.1. Speech Resynthesis and Voice Conversion: ZipCodec achieves the strongest overall speech resynthesis performance among streaming codecs in English and multilingual settings, improving perceptual quality, intelligibility, and speaker fidelity.It operates at 6.25 Hz versus 50 Hz for FocalCodec-Stream and at least 12.5 Hz for other streaming baselines.
- 4.1. Speech Resynthesis and Voice Conversion: ZipCodec achieves the highest perceptual quality among streaming codecs in voice conversion while maintaining strong speaker fidelity and competitive intelligibility.FocalCodec-Stream remains second-best for speaker fidelity, while FocalCodec-Stream and PAST achieve lower dWER.
- 4.2. Downstream Tasks: ZipCodec is best among streaming codecs on speaker identification, emotion recognition, keyword spotting, and intent classification, while ranking second on automatic speech recognition.It consistently improves over FocalCodec-Stream across all five discriminative tasks.
- 4.2. Downstream Tasks: ZipCodec achieves the strongest overall streaming performance on speech enhancement and separation, outperforming FocalCodec-Stream and the non-streaming baseline on separation.Its sequences are 8 times shorter than the 50 Hz representations used by FocalCodec-Stream.
- 4.3. Streaming Efficiency: Despite 842M parameters, ZipCodec supports real-time single-stream CPU inference with RTF 1.33 and p99 latency below the 160 ms codec frame duration.GPU inference remains real-time at batch size 16.
5. CONCLUSION
ZipCodec combines 6.25 Hz, 0.80 kbps streaming with strong reconstruction and representation quality, improving over streaming baselines and enabling real-time consumer-CPU inference.
- 5. CONCLUSION: ZipCodec operates at 6.25 Hz and 0.80 kbps with 160 ms theoretical latency while achieving strong reconstruction and representation quality.It substantially reduces frame rate compared with existing streaming codecs.
- 5. CONCLUSION: ZipCodec consistently improves over streaming baselines across reconstruction and downstream tasks while enabling real-time inference on a consumer-grade CPU.These results hold despite the codec's large model size.