Source-linked AI summary
FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks
Luca Della Libera, Francesco Paissan, Cem Subakan, Mirco Ravanelli
TL;DR
Speech codecs must preserve semantic and acoustic information while operating at low bitrates, but existing approaches can lose one kind of information or rely on complex multi-codebook designs. FocalCodec uses focal modulation and a single binary codebook. It delivers competitive speech resynthesis and voice conversion across diverse conditions, while preserving information useful for downstream tasks.
Problem
Speech codecs must retain acoustic and semantic information at low token rates, while existing approaches may lose one type of information or use complex multi-codebook designs.
Source: §1 · p. 1Method
FocalCodec uses a focal-modulation architecture and a single binary codebook to compress speech at 0.16 to 0.65 kbps.
Source: §1 · p. 2, §3.1 · p. 3Results
FocalCodec delivers competitive speech resynthesis and voice conversion at low bitrates, with reported performance across clean, noisy, and multilingual speech.
Source: §4.3 · p. 7, §4.4 · p. 8, §5 · p. 10Takeaways & Limitations
FocalCodec preserves semantic and acoustic information in discrete representations for downstream tasks, including generative speech modeling.
Source: §1 · p. 2, §5 · p. 10The pipeline beyond the WavLM encoder was trained on only a few hundred hours of clean English speech, less data and diversity than competing approaches.
Source: A Limitations · p. 16
Abstract
from arXiv · showhide
Large language models have revolutionized natural language processing through self-supervised pretraining on massive datasets. Inspired by this success, researchers have explored adapting these methods to speech by discretizing continuous audio into tokens using neural audio codecs. However, existing approaches face limitations, including high bitrates, the loss of either semantic or acoustic information, and the reliance on multi-codebook designs when trying to capture both, which increases architectural complexity for downstream tasks. To address these challenges, we introduce FocalCodec, an efficient low-bitrate codec based on focal modulation that utilizes a single binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec delivers competitive performance in speech resynthesis and voice conversion at lower bitrates than the current state-of-the-art, while effectively handling multilingual speech and noisy environments. Evaluation on downstream tasks shows that FocalCodec successfully preserves sufficient semantic and acoustic information, while also being well-suited for generative modeling. Demo samples and code are available at https://lucadellalib.github.io/focalcodec-web/.
1 Introduction
Speech tokenization draws on the success of language-model approaches, but codecs must preserve acoustic and semantic information at low token rates. FocalCodec addresses this challenge with a focal-modulation codec using a single binary codebook.
1 Introduction
- Neural audio codecs compress speech into tokens for downstream models, which need acoustic and semantic information while keeping token rates low.Longer sequences make long-term dependencies harder to capture and increase computational costs.Source: §1 · p. 1
- Existing codecs struggle to combine low bitrate, reconstruction quality, and useful downstream representations without multi-codebook complexity or other elaborate techniques.Acoustic codecs often lack strong semantic representations; hybrid codecs may rely on disentanglement, distillation, or supervised fine-tuning.Source: §1 · p. 1
- FocalCodec uses a compressor–quantizer–decompressor architecture and a single binary codebook to compress speech at 0.16 to 0.65 kbps.The encoder extracts acoustic and semantic features, which are compressed, binary quantized, projected back, and decoded into a waveform.Source: §1 · p. 2, Figure 1 · p. 2
- Its focal modulation-based architecture is presented as an efficient, scalable approach to speech tokenization.Source: §1 · p. 2
- Evaluations cover reconstruction quality and downstream tasks, with the codec positioned for both discriminative and generative speech modeling.Source: §1 · p. 2
2 Related Work
Prior speech codecs trade off reconstruction, semantic content, and architectural simplicity. FocalCodec is positioned as a self-supervised hybrid alternative that combines semantic and acoustic information in one codebook.
Acoustic Codecs
- Acoustic codecs prioritize high-fidelity reconstruction, while multi-scale RVQ reduces bitrate but adds token-sequence complexity for downstream applications.Single-codebook designs offer a simpler, efficient alternative; FocalCodec follows this direction with focal modulation and a pretrained self-supervised encoder.Source: §2 · p. 2
Semantic Codecs
- Semantic codecs capture linguistic and content information but can discard acoustic detail, reducing speaker fidelity in direct resynthesis.Some methods add continuous embeddings for prosody and speaker traits, but this undermines unified semantic and acoustic tokenization.Source: §2 · pp. 2–3
Hybrid Codecs
- Hybrid codecs combine semantic and acoustic features, often using multiple codebooks, separate encoders, distillation, or supervised fine-tuning.FocalCodec instead uses self-supervised learning to compress both kinds of information into a single codebook.Source: §2 · p. 3
3 FocalCodec
FocalCodec combines a focal-modulation compressor–quantizer–decompressor with a single binary codebook. Its two-stage training separates representation reconstruction from audio resynthesis while retaining a shared decoder at inference.
3.1 Architecture
- The encoder uses the first six layers of WavLM-large, while a focal compressor maps its representations into a compact latent space.The compressor can also downsample the sequence over time.Source: §3.1 · p. 3
- Focal modulation replaces self-attention by aggregating global context before using it to modulate local token interactions.Hierarchical depth-wise convolutions capture short-to-long-range context, with pooling adding global information.Source: §3.1 · pp. 3–4
- FocalCodec uses one quantizer with binary spherical quantization, avoiding hierarchical codebook designs in downstream models.BSQ is a lookup-free method with an implicit codebook.Source: §3.1 · p. 4
- Binary spherical quantization normalizes latent vectors before binary quantization and uses a straight-through estimator for differentiability.The implicit codebook is parameter-free, and its size is tied to the latent dimension.Source: §3.1 · p. 4
- The decoder uses an asymmetric design that allocates about five times more parameters to the encoder, with Vocos for waveform reconstruction.The discriminator uses multi-period and multi-scale components, following the simpler HiFi-GAN setup for speech.Source: §3.1 · p. 5
3.2 Training
- Training first jointly fits the compressor, quantizer, and decompressor to reconstruct frozen encoder features, then trains the decoder for audio resynthesis.At inference, the same decoder processes dequantized features; the decoupled setup requires no additional fine-tuning and is intended to preserve semantic and acoustic information.Source: §3.2 · p. 5
4 Experiments
The experiments evaluate three token-rate variants against semantic and hybrid codecs across speech resynthesis, voice conversion, downstream tasks, and component ablations. FocalCodec@50 performs strongly in resynthesis and downstream evaluation, while lower-rate variants remain competitive in some settings and @25 leads TTS.
4.1 FocalCodec
- FocalCodec is trained on LibriTTS at 16 kHz in 50, 25, and 12.5 Hz variants, using an 8192-entry codebook and different compressor downsampling factors.The corresponding downsampling patterns are mirrored in the decompressor for upsampling.Source: §4.1 · p. 5
4.2 Baselines
- The baseline comparison includes a reimplemented WavLM6-KM semantic codec and hybrid codecs evaluated from their official checkpoints.The authors prioritize hybrid codecs because FocalCodec is also hybrid, while semantic codecs typically have weaker reconstruction quality or require higher bitrates to compete.Source: Table 1 · p. 6, §4.2 · p. 6
4.3 Speech Resynthesis
- FocalCodec@50 delivers the best clean-speech trade-off, lowest dWER in clean and multilingual speech, and lowest dWER on VoiceBank and Libri1Mix noisy resynthesis.Lower-rate variants show some degradation, particularly in multilingual speech; FocalCodec@12.5 remains competitive with higher-bitrate baselines at 0.16 kbps.Source: Table 2 · p. 7, §4.3 · p. 7
4.4 Voice Conversion
- In one-shot voice conversion, FocalCodec achieves the highest speaker similarity while maintaining good intelligibility, outperforming hybrid codecs designed for explicit semantic-acoustic disentanglement.The evaluation uses VCTK.Source: Table 3 · p. 8, §4.4 · p. 8
4.5 Downstream Tasks
- Across discriminative tasks, FocalCodec@50 is competitive overall and on par with the best in SER; it leads SE dWER and SS baselines, while @25 achieves best overall TTS.The absolute speech-separation performance remains far from practical utility; the authors trained TTS on only 460 hours of speech.Source: Table 4 · p. 9, §4.5 · p. 9
4.6 Ablation Studies
- Replacing focal modulation blocks with Conformer, AMP, or linear layers causes the largest performance drop, particularly in dWER and Sim.The ablations use a smaller FocalCodec variant because of limited computational resources.Source: Table 5 · p. 10, §4.6 · p. 10
5 Conclusions
FocalCodec delivers competitive speech resynthesis and voice conversion at low and ultra-low bitrates, including multilingual and noisy speech. It preserves semantic and acoustic information for downstream tasks.
5 Conclusions
- FocalCodec delivers competitive speech resynthesis and voice conversion at low and ultra-low bitrates while remaining robust across multilingual and noisy speech.The conclusion also describes robustness across diverse conditions.Source: §5 · p. 10
- FocalCodec effectively preserves both semantic and acoustic information in its discrete representations for downstream tasks.Source: §5 · p. 10
A Limitations
FocalCodec is undertrained relative to competing approaches, and the authors identify broader training data as a possible route to improved quality, robustness, and versatility. The societal impact discussion notes potential bandwidth and storage benefits alongside misuse risks from voice conversion.
A Limitations
- The pipeline beyond the WavLM encoder was trained on only a few hundred hours of clean English speech, versus substantially larger and more diverse datasets for named competing methods.The authors suggest broader data and domains could further improve quality, robustness, and versatility.Source: A Limitations · p. 16
B Societal Impact
- Ultra-low-bitrate codecs may reduce communication bandwidth and storage needs, while voice conversion could be misused for cloning, impersonation, or deceptive audio.The authors encourage responsible use and further detection and authentication research.Source: B Societal Impact · p. 16
C Datasets
The datasets span English, multilingual, noisy, and task-specific speech resources. The baseline appendix also describes codec references and a WavLM6-KM reproduction.
C Datasets
- LibriSpeech and LibriTTS provide English read speech and text-to-speech research data, respectively.LibriSpeech is derived from audiobooks; LibriTTS is constructed from the same source.Source: C Datasets · p. 16
- Multilingual LibriSpeech supplies multilingual speech, while VoiceBank and LibriMix cover noisy speech and speech mixtures.VoiceBank includes noise-conditioned training and test utterances; LibriMix mixes LibriSpeech utterances with WHAM! noise.Source: C Datasets · p. 16
- VCTK provides English recordings from speakers with varied accents, and IEMOCAP provides dialogues with emotion labels.Source: C Datasets · p. 16
D Baselines
- Table 6 provides additional information about baseline codecs, and the WavLM6-KM reproduction uses 512-centroid k-means quantization of WavLM-large layer-6 representations.Its reproduced pipeline also trains a dequantizer and HiFi-GAN decoder.Source: D Baselines · pp. 16–17, Table 6 · p. 17
E Hyperparameters and Training Details
The training details describe the focal compressor-quantizer-decompressor stage, a separate waveform decoder, and a smaller configuration for ablations.
E.1 FocalCodec
- The compressor maps 1024-dimensional WavLM features through three focal downscaling blocks to 13-dimensional latents quantized by an 8192-code binary spherical codebook.It is trained on full LibriTTS utterances, and the decompressor mirrors its structure with focal upscaling blocks.Source: §E.1 · p. 17
- The decoder uses eight ConvNeXt blocks and is trained on LibriTTS train-clean-100, with training continuing until perceived audio quality stops improving around 3M steps.The decoder uses 7040-sample chunks and a HiFi-GAN-style discriminator architecture.Source: §E.1 · p. 17
- The ablation variant reduces the focal-block hidden sizes and codebook to 1024 codes, and uses a HiFi-GAN-V1 decoder trained on LibriSpeech train-clean-100.Source: §E.1 · p. 17
E.2 Downstream Tasks
Downstream evaluations use task-specific probes and generative models to assess information retained in codec tokens. The setups vary across recognition, enhancement, separation, and speech synthesis.
Automatic Speech Recognition (ASR)
- ASR uses a 2-layer, 512-dimensional BiLSTM with a CTC head; 12.5-Hz tokens make character training infeasible, and multi-codebook inputs use learned weighted embedding sums.The best result is reported across character and BPE vocabularies of sizes 250, 500, and 1,000.Source: §E.2 · pp. 17–18
Speaker Identification (SI) and Speech Emotion Recognition (SER)
- Speaker identification aggregates BiLSTM outputs with statistics pooling, while emotion recognition uses the same setup with a different number of output classes.Both setups otherwise mirror ASR.Source: §E.2 · p. 18
Speech Enhancement (SE) and Speech Separation (SS)
- Speech enhancement uses a 6-layer Conformer, while speech separation adds permutation-invariant training and doubles the output heads to predict two sources.Both use cross-entropy training; multi-codebook codecs use weighted input embeddings and independent output heads.Source: §E.2 · p. 18
Text-To-Speech (TTS)
- Text-to-speech uses an autoregressive Llama 3 decoder conditioned on a pooled speaker embedding prepended to the text embeddings.Training predicts speech tokens only; inference uses top-p sampling with p = 0.9 and temperature 1.0.Source: §E.2 · p. 18
Training Details
- Across tasks, training uses AdamW with batch size 16, learning rate 0.0001, and dropout 0.1, with learning-rate reduction and early stopping based on validation loss.Gradients are clipped to a maximum L2 norm of 0.01.Source: §E.2 · p. 18
F Implementation and Hardware
The implementation uses SpeechBrain on a single GPU. In clean-speech resynthesis, FocalCodec compares favorably with TS3-Codec, including at a lower bitrate for some metrics.
F Implementation and Hardware
- The experimental evaluation was implemented in Python with SpeechBrain, and each model was trained on one V100 or A100 GPU depending on cluster availability.The available GPU memory configurations were V100 GPUs with 16 or 32 GB, or A100 GPUs with 40 GB.Source: F Implementation and Hardware · p. 18
G.1 Comparison to TS3-Codec
- FocalCodec@50 surpasses TS3-Codec across all evaluated metrics, while FocalCodec@25 achieves superior UTMOS and dWER at a significantly lower bitrate.Table 7 covers clean speech resynthesis on LibriSpeech test-clean.Source: Table 7 · p. 18, §G.1 · p. 18
G.2 Subjective Evaluation
In a subjective listening evaluation, FocalCodec@50 performed comparably to leading codecs at very low bitrates. The paper also evaluates chunked and causal streaming configurations, with the latter reaching 80-millisecond latency.
G.2 Subjective Evaluation
- The MUSHRA test used 10 LibriSpeech test-clean reconstructions, selected a subset of baselines, and retained 33 valid submissions from 40 participants.Submissions were filtered out if listeners spent less than 60 seconds on any reconstruction.Source: §G.2 · p. 19
- 33 valid listeners rated 10 reconstructions; FocalCodec@50 outperformed most baselines and remained comparable to BigCodec and Stable Codec at extremely low bitrates.The evaluation used MUSHRA ratings; submissions with less than 60 seconds spent on any reconstruction were excluded.Source: Figure 2 · p. 19, §G.2 · p. 19
G.3 Streaming Inference
- Chunked inference used 500-millisecond chunks, about 4% overlap, and 3 seconds of left context; performance remained acceptable but latency was too high for strict real-time use.Reconstructed chunks were stitched with overlap-add and linear fade-in/fade-out.Source: §G.3 · p. 19, Table 8 · p. 19
- Causal replacements, WavLM feature distillation, 24-kHz decoding, increased model capacity, and additional data yielded 80-millisecond latency with a real-time factor suitable for consumer-grade GPUs.The codec was made streamable through chunked inference despite being non-causal.Source: §G.3 · p. 19, Table 8 · p. 19
G.4 Mel-Spectrogram Analysis
Mel-spectrogram examples show near-ground-truth reconstruction on clean LibriSpeech and different denoising behavior on noisy Libri1Mix.
G.4 Mel-Spectrogram Analysis
- On LibriSpeech, reconstructed speech is almost indistinguishable from ground truth in the examples.Figure 3 shows LibriSpeech on the left and Libri1Mix on the right.Source: Figure 3 · p. 20, §G.4 · p. 20
- On noisy Libri1Mix, BigCodec attempts to reconstruct noise, whereas Stable Codec and FocalCodec perform basic denoising.The comparison uses the top three performing codecs.Source: Figure 3 · p. 20, §G.4 · p. 20
- FocalCodec assigns more energy to speech-frequency bands than the clean original, while Stable Codec adds lower-spectrogram artifacts and static noise that degrade intelligibility.Source: Figure 3 · p. 20, §G.4 · p. 20