Source-linked AI summary
FocalCodec-Stream: Streaming Low-Bitrate Speech Coding via Causal Distillation
Luca Della Libera, Cem Subakan, Mirco Ravanelli
TL;DR
Most neural audio codecs are non-streamable, despite the need for low-bitrate acoustic and semantic tokens in real-time audio applications. FocalCodec-Stream combines causal distillation with architectural changes to produce a single-codebook streaming codec. At comparable bitrates, it outperforms popular streamable baselines while retaining semantic and acoustic information.
Problem
Most codecs are designed for offline processing and depend on long future-context windows, while streamable alternatives can compromise bitrate, codebook simplicity, semantic information, or downstream representations.
Source: §1 · p. 1Method
FocalCodec-Stream adapts WavLM through multi-stage causal distillation and targeted architectural changes, including a lightweight refiner, for single-binary-codebook streaming.
Source: §1 · p. 1Results
Takeaways & Limitations
The codec achieves a favorable balance of reconstruction quality, downstream-task performance, latency, and efficiency while preserving semantic and acoustic information.
Source: §5 · p. 4A performance gap with full-context FocalCodec@50 remains, especially at lower bitrates, under the stricter constraints of real-time streaming.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
Neural audio codecs are a fundamental component of modern generative audio pipelines. Although recent codecs achieve strong low-bitrate reconstruction and provide powerful representations for downstream tasks, most are non-streamable, limiting their use in real-time applications. We present FocalCodec-Stream, a hybrid codec based on focal modulation that compresses speech into a single binary codebook at 0.55 - 0.80 kbps with a theoretical latency of 80 ms. Our approach combines multi-stage causal distillation of WavLM with targeted architectural improvements, including a lightweight refiner module that enhances quality under latency constraints. Experiments show that FocalCodec-Stream outperforms existing streamable codecs at comparable bitrates, while preserving both semantic and acoustic information. The result is a favorable trade-off between reconstruction quality, downstream task performance, latency, and efficiency. Code and checkpoints will be released at https://github.com/lucadellalib/focalcodec.
1. INTRODUCTION
Neural audio codecs support generative audio by providing compact tokens that retain acoustic and semantic information, but most codecs are not streamable. FocalCodec-Stream extends FocalCodec with a single-codebook streaming design and multi-stage causal distillation.
1. INTRODUCTION
- Neural audio codecs provide discrete audio tokens for text-conditioned audio generation and speech language models, which require acoustic and semantic information at low bitrates.Unlike transmission, effective SLM training requires much lower bitrates than the 12 kbps codecs already noted to outperform traditional codecs.Source: §1 · p. 1
- Most codecs depend on long future-context windows, limiting their suitability for real-time speech assistants, interactive dialogue, and low-latency generation.Existing streamable codecs often trade off bitrate, single-codebook design, semantic information, or downstream-task representations.Source: §1 · p. 1
- FocalCodec [22] uses WavLM, focal modulation, and binary spherical quantization in a single codebook, but does not support streaming.This paper extends that codec to the streaming setting.Source: §1 · p. 1
- FocalCodec-Stream uses a single binary codebook at 0.55 - 0.80 kbps and has a theoretical latency of 80 ms.Its approach combines multi-stage causal distillation of WavLM with architectural changes, including a lightweight refiner to mitigate quality degradation under latency constraints.Source: §1 · p. 1
2. FOCALCODEC-STREAM
FocalCodec-Stream makes the FocalCodec architecture causal and adapts WavLM through multi-stage distillation. A lightweight refiner helps align the causal codec representations with the full-context features used by the decoder.
2.1. Codec architecture
- The encoder uses causal convolutions and sliding-window chunked attention; compressor and decompressor use causal convolutions and causal pooling, while the decoder uses causal blocks and a projection-and-flattening waveform reconstruction.The 80 ms attention chunks allow up to three frames of lookahead; most other modules are strictly causal and incur 20 ms latency.Source: §2.1 · pp. 1–2
- A residual chunk-wise feed-forward refiner after the decompressor uses available latency to align representations with WavLM features.It improves perceptual quality at the target latency with minimal impact on inference speed.Source: §2.1 · p. 2
2.2. Causal distillation
- Stage 1 distills the positional embedding because its 2.56 s receptive field exceeds the 80 ms latency budget; the causal variant is trained to approximate the full-context version with an L2 loss.The procedure uses a single convolutional layer with 8.4M parameters, less than 10% of the encoder size.Source: §2.2 · p. 2
- Stage 2 converts attention and the convolutional feature extractor to causal form, distilling each of six layers against the teacher with greater weight on deeper layers.The reversed linear loss weights range from 1.0 for layer 6 to 0.9 for layer 5 and 0.8 for layer 4, giving earlier layers more freedom to adapt.Source: §2.2 · p. 2
- Stage 3 trains the causal compressor–quantizer–decompressor on the distilled encoder and trains the causal decoder on full-context WavLM representations.An L2 loss aligns the distilled and decompressed representations.Source: §2.2 · p. 2
- Stage 4 jointly fine-tunes the encoder through refiner against teacher outputs to address the distribution shift between the full-context-trained decoder and causal decompressor.This optional stage considerably improves reconstruction quality and lets the encoder produce features better suited to decompressor reconstruction.Source: §2.2 · p. 2
3. EXPERIMENTAL SETUP
Experiments use three 50 Hz codec variants with different codebook sizes and progressively scale training data across distillation stages. The decoder is trained at 24 kHz to provide super-resolution relative to 16 kHz.
3. EXPERIMENTAL SETUP
- The 50 Hz model variants use codebook sizes of 2048, 4096, and 65,536, with compressor and decompressor focal layers set to hidden size 1024 and focal window and factor 14 and 4.The decoder uses hidden size 1024, feed-forward dimension 2048, and upsampling factor 480 for 24 kHz reconstruction rather than 16 kHz; sliding-window operations use 10.24 s of past context.Source: §3 · p. 3
- Training data scales from LibriTTS (585 hours) in Stage 1, to Libri-Light-medium (5k hours) in Stages 2 and 3, and full Libri-Light (60k hours) in Stage 4.Stage 4 uses 30 s chunks; Stages 2 and 3 use 15 s chunks, while the decoder uses the clean-100 LibriTTS split at 24 kHz with 3 s chunks.Source: §3 · p. 3
4. RESULTS
The results evaluate speech resynthesis and voice conversion alongside discriminative and generative downstream tasks, then ablate the refiner and final fine-tuning stage. FocalCodec-Stream performs strongly across these evaluations, with quality trade-offs under streaming constraints.
4.1. Speech resynthesis and voice conversion
- On LibriSpeech, FocalCodec-Stream approaches full-context FocalCodec in UTMOS and beats all baselines in intelligibility; its 50-65k variant has the best dWER and speaker similarity.For multilingual MLS speech, its variants outperform acoustic codecs and remain competitive with Mimi6; 50-65k offers the best intelligibility–speaker-fidelity trade-off.Source: §4.1 · p. 3, Table 2 · p. 4
4.2. Downstream tasks
- Across discriminative tasks, FocalCodec-Stream ranks among the top codecs at lower bitrates, with roughly 17% WER on ASR for all variants.The 65k variant has the best speaker-identification error rate, the 4k variant leads on speech emotion recognition, and the models outperform prior baselines on keyword spotting and intent classification.Source: §4.2 · p. 4, Table 3 · p. 4
- For speech enhancement, all three variants achieve high DNSMOS and competitive speaker similarity; their dWER beats all baselines except PAST.For speech separation, the 65k variant substantially outperforms all baselines and rivals the non-streaming model across metrics.Source: §4.2 · p. 4, Table 3 · p. 4
- A performance gap from full-context FocalCodec@50 remains, especially at lower bitrates, under the stricter constraints of real-time streaming.The authors describe the resulting balance across reconstruction quality, downstream performance, latency, and efficiency as favorable.Source: §4.2 · p. 4
4.3. Ablation studies
- The full model achieves the best UTMOS, dWER, and similarity; removing the refiner degrades performance, while omitting stage-4 fine-tuning has an even stronger effect.The reported explanation is that the refiner mitigates distribution shift between causal and full-context features; together, the ablations support the progressive multi-stage framework under low-latency constraints.Source: §4.3 · p. 4, Table 4 · p. 4
5. CONCLUSION
FocalCodec-Stream extends FocalCodec with a causal single-codebook design and multi-stage distillation. It operates at 0.55–0.80 kbps with 80 ms theoretical latency and outperforms popular streamable baselines at comparable bitrates.
5. CONCLUSION
- At comparable bitrates, FocalCodec-Stream outperforms popular streamable baselines while preserving semantic and acoustic information.The conclusion reports strong reconstruction quality alongside these outcomes.Source: §5 · p. 4
- FocalCodec-Stream combines a causal single-codebook design with a multi-stage distillation strategy.Source: §5 · p. 4
- At 0.55–0.80 kbps, FocalCodec-Stream has an 80 ms theoretical latency.Source: §5 · p. 4
- Future work will scale to larger datasets, explore autoregressive modeling, and further optimize for resource-constrained settings.Source: §5 · p. 4