Source-linked AI summary
UniStream: Multi-Expert Residual Vector Quantization for 48 kHz Causal Streaming Audio Coding
Mingyu Zhao, Zhiyong Wu
TL;DR
Causal audio codecs remain challenging to make efficient and robust across speech, music, and environmental audio, while shared RVQ codebooks and transmitted MoE routing can limit specialization or complicate streaming. UniStream combines deterministic multi-expert residual quantization with a training-only flow-matching regularizer; reported evaluations identify ME-RVQ as the primary source of improvement and show competitive lower-rate operation alongside results approaching higher-rate references.
Problem
Designing a codec that is simultaneously causal, efficient, and robust across speech, music, and environmental audio remains challenging; shared RVQ codebooks and transmitted MoE routing can constrain specialization or complicate streaming.
Source: §1 · p. 2Method
UniStream combines ME-RVQ with deterministic decoder-side routing and an auxiliary OT-CFM objective used only to regularize quantized latents during training.
Source: §1 · p. 2Results
ME-RVQ is the primary source of improvement: Top-1 is competitive at 12 kbps, while Top-2 approaches higher-rate references under full-band perceptual evaluation.
Source: §1 · p. 2, §5 · p. 13Takeaways & Limitations
ME-RVQ provides the main reconstruction improvement, while OT-CFM adds speech perceptual gains without inference-time overhead.
Source: §5 · pp. 12–13
Abstract
from arXiv · showhide
We present UniStream, a fully causal 48 kHz neural audio codec for streaming speech, music, and environmental sounds. At its core is Multi-Expert Residual Vector Quantization (ME-RVQ), which replaces the single shared codebook in each residual quantization layer with four expert codebooks controlled by a deterministic Top-K router. Because routing decisions are derived solely from previously decoded quantized states, the decoder can reproduce the selected experts without transmitting expert identifiers, thereby expanding quantization capacity while adding 5.5M parameters. We further introduce an auxiliary Optimal Transport Conditional Flow Matching (OT-CFM) objective to regularize the quantized latent space during training. The flow module is removed entirely at inference and therefore incurs no runtime overhead. UniStream supports a 12 kbps Top-1 mode and a 22.5 kbps Top-2 mode within a causal 48 kHz encoder-decoder framework, while achieving real-time GPU inference. To complement narrow-band speech metrics, we report 48 kHz ViSQOL audio mode, ViSQOL speech mode, standard VGGish-FAD, DNSMOS P.835, and higher-rate reference comparisons with Opus and EnCodec. At 12 kbps, UniStream-Top1 achieves PESQ and UTMOS scores comparable to EnCodec while reducing speech Mel-D from 13.07 to 8.21. At 22.5 kbps, UniStream-Top2 achieves a ViSQOL speech-mode score of 4.67 and an environmental audio-mode score of 3.96, exceeding all evaluated systems operating at 12 kbps or below in the latter setting. It also comes within 0.03 MOS-LQO of Opus at 24 kbps on speech in ViSQOL audio mode. Ablation studies confirm that ME-RVQ is the primary source of quality improvement, whereas OT-CFM provides perceptual gains on speech with a mild trade-off in spectral distortion.
1 Introduction
UniStream addresses the challenge of causal, efficient audio coding across speech, music, and environmental sounds by changing the RVQ quantization architecture and adding a training-only flow objective. Its reported evaluations find ME-RVQ is the primary source of improvement, with Top-1 competitive at 12 kbps and Top-2 approaching higher-rate references.
1 Introduction
- Shared RVQ codebooks must model heterogeneous acoustic structures, while prior MoE codecs may require transmitted routing information that complicates streaming.Speech formants, musical harmonics, and environmental textures have different temporal and spectral characteristics.Source: §1 · p. 2
- ME-RVQ replaces each shared acoustic RVQ codebook with expert codebooks and a learned Top-K router whose selections the decoder can reproduce without expert identifiers.The design expands quantization capacity while adding 5.5M parameters.Source: §1 · p. 2
- OT-CFM regularizes the quantized latent space during training, then is removed so it adds no runtime overhead.The codec supports a 12 kbps Top-1 mode and a 22.5 kbps Top-2 mode.Source: §1 · pp. 2–3
- ME-RVQ is the primary source of improvement: Top-1 provides a competitive 12 kbps operating point, while Top-2 approaches higher-rate references under full-band perceptual evaluation.The evaluation covers conventional and full-band perceptual metrics across speech, music, and environmental audio.Source: §1 · pp. 2–3
- The evaluation reports full-band perceptual measures alongside speech metrics and compares UniStream with higher-rate Opus and EnCodec references.The reported measures include ViSQOL, VGGish-FAD, and DNSMOS P.835.Source: §1 · p. 3
2 Related Work
Related work establishes neural audio codecs and expert-based quantization as foundations for UniStream, while identifying limitations around causal, cross-domain coding and routing overhead. UniStream places adaptive expert routing in the causal quantizer and uses flow matching only as a training regularizer.
2.1 Neural Audio Codecs
- UniStream modifies the quantization architecture for causal 48 kHz general-audio coding, with decoder-side expert routing that requires no transmitted identifiers.Earlier codec work includes RVQ-based frameworks and systems exploring multi-scale quantization, semantic–acoustic representations, compact tokenization, and flexible sample-rate coding.Source: §2.1 · p. 3
2.2 Quantization and Mixture-of-Experts Modeling
- ME-RVQ derives routing from the cumulative quantized reconstruction shared by encoder and decoder, avoiding the explicit routing information that prior residual-expert methods may require.Unlike post-hoc bitrate allocation, UniStream integrates adaptive routing into the causal quantization architecture.Source: §2.2 · p. 3
2.3 Flow-Based Audio Coding and Training Regularization
- UniStream uses OT-CFM only during training to encourage quantized latents to retain information about continuous encoder outputs, then removes the flow module at inference.This retains the training signal without runtime overhead from iterative flow-model inference.Source: §2.3 · p. 4
3 Proposed Method
UniStream combines a causal 48 kHz encoder–quantizer–decoder with multi-expert residual quantization and a flow-matching training regularizer. Its design supports causal streaming while expanding quantization capacity and avoiding inference-time flow overhead.
3.1 System Overview
- The causal encoder maps 48 kHz waveforms to continuous latents, which are quantized and decoded by a symmetric causal CNN architecture.Four strided blocks yield a downsampling factor of 320 and a latent frame rate of 150 Hz; all inference modules are fully causal.Source: §3.1 · pp. 4–5
3.2 Multi-Expert Residual Vector Quantization
- ME-RVQ replaces shared residual codebooks with expert codebooks across seven residual layers, following a shared base layer, to increase quantization capacity.The router selects Top-K experts per frame and renormalizes their weights; the average entry utilization across expert codebooks is 87.3%.Source: Split RVQ Structure · p. 5, §3.2 · p. 5, MoE Quantizer Layer · pp. 5–6
3.3 OT-CFM Training Regularizer
- OT-CFM conditions velocity prediction on the quantized latent to encourage it to retain information about the continuous encoder output.The flow module is used only during training and removed at inference, adding no runtime overhead or streaming latency.Source: §3.3 · p. 7
3.4 Training Objective
- Training combines reconstruction, adversarial, quantization, routing, and flow-regularization losses.Adversarial training starts after 10k steps, while flow matching starts after 5k steps once the codebooks have stabilized.Source: Routing Regularization · p. 6, §3.4 · p. 8
4 Experiments
Experiments compare UniStream across conventional and full-band perceptual measures and ablate its two training components. Results show strong Top-2 full-band scores, complementary component effects, and near-real-time inference.
4.1 Experimental Setup
- Training mixes LibriSpeech, MTG-Jamendo, and FSD50K at probabilities of 0.5, 0.3, and 0.2; evaluation uses fixed subsets of 200, 100, and 94 items.Neural baselines use official checkpoints or recommended configurations rather than being retrained on the mixed-domain corpus; metrics include conventional speech and full-band perceptual measures.Source: §4.1 · pp. 8–9
4.2 Conventional Objective Metrics
- At 12 kbps, UniStream-Top1 reduces speech Mel-D from 13.07 to 8.21 versus EnCodec, with comparable PESQ and UTMOS but lower STOI (0.826).Environmental Mel-D also decreases from 13.18 to 10.16.Source: §4.2 · p. 9, Table 1 · p. 10
4.3 Full-Band ViSQOL Evaluation
- At 22.5 kbps, UniStream-Top2 reaches 4.67 ViSQOL speech-mode and 3.96 environmental audio-mode, exceeding systems at 12 kbps or below on environmental audio.It is within 0.03 MOS-LQO of Opus at 24 kbps on speech in ViSQOL audio mode.Source: Table 2 · p. 10, §4.3 · p. 10
4.4 VGGish-FAD Evaluation
- UniStream-Top2 achieves speech FAD of 0.27 and environmental FAD of 0.44, while its music FAD of 0.68 remains higher than EnCodec, DAC, and SNAC.Lower VGGish-FAD indicates closer distributional alignment with the reference audio.Source: §4.4 · pp. 10–11, Table 3 · p. 11
4.5 DNSMOS P.835 Speech Evaluation
- UniStream-Top2's DNSMOS P.835 scores of 3.55 SIG, 3.89 BAK, and 3.20 OVR closely match EnCodec at 24 kbps.DAC and Mimi score slightly higher but are reported as reference systems with different configurations and bitrates.Source: Table 4 · p. 11, §4.5 · p. 11
4.6 Ablation Study
- Removing ME-RVQ drops PESQ from 3.41 to 3.20 and UTMOS from 3.87 to 3.80; removing OT-CFM slightly lowers speech metrics but marginally improves Mel-D across domains.The ablation identifies ME-RVQ as the primary contributor to overall quality improvement and OT-CFM as an auxiliary speech-perceptual regularizer.Source: §4.6 · pp. 11–12, Table 5 · p. 12
4.7 Efficiency
- Inference uses 47.9M parameters (75.6M during training); A100 RTF is 0.144 for real-time operation, while Xeon CPU RTF 1.05 is approximately real-time.The latent-frame interval is a lower bound on streaming latency, not the complete end-to-end algorithmic latency.Source: §4.7 · p. 12
5 Conclusion
UniStream is a fully causal 48 kHz codec whose ME-RVQ enables deterministic decoder-side expert routing without expert-ID signaling. Its Top-1 and Top-2 modes show competitive spectral reconstruction and strong full-band results, while training-only OT-CFM adds modest speech-perceptual gains without inference overhead.
5 Conclusion
- ME-RVQ expands RVQ capacity through expert codebooks while deterministic decoder-side routing avoids expert-ID signaling.The module adds 5.5M parameters.Source: §5 · p. 12
- Training-only OT-CFM provides modest perceptual gains on speech without inference-time overhead.Source: §5 · pp. 12–13
- Top-1 provides a competitive 12 kbps operating point with improved spectral reconstruction, while Top-2 approaches higher-rate references under full-band evaluation and performs strongly on environmental audio.Source: §5 · p. 13
- Future work will investigate subjective listening tests, entropy coding, and deployment on mobile and edge hardware.Source: §5 · p. 13