PaperlayerView exact revision on arXiv

Source-linked AI summary

BiMTokenizer: Preserving Semantic-Acoustic Balance in Low-Bitrate Speech Tokenization via Bidirectional State-Space Modeling

Xin Zhang, Lin Li, Chuanbo Liu, Jianquan Liu, Kong Aik Lee

arXiv:2609.00562v1cs.SD

TL;DR

Low-bitrate speech codecs must balance semantic preservation with acoustic fidelity, a challenge that has encouraged architecturally costly dual-tower designs. BiMTokenizer instead uses a single tower with bidirectional state-space modeling and Residual Spherical Leech Quantization, achieving strong reconstruction and semantic results with fewer parameters. Its offline requirement, large fixed vocabulary, and heterogeneous CNN–Mamba design remain scope boundaries.

  • Problem

    Low-bitrate speech codecs must preserve semantic content and acoustic fidelity for Speech LLMs, but these objectives compete and have driven recent dual-tower designs.

  • Method

    BiMTokenizer is a roughly 1.1 kbps single-tower codec combining a bidirectional Mamba backbone, Residual Spherical Leech Quantization, and training-only frozen semantic supervision.

  • Results

    BiMTokenizer matches or surpasses recent dual-tower codecs on reconstruction and semantic benchmarks while using far fewer parameters.

  • Takeaways & Limitations

    Careful redesign of a single-tower codec’s shared backbone and bottleneck remains a viable path for low-bitrate speech tokenization.

  • Takeaways & Limitations

    The bidirectional backbone requires future frames, limiting BiMTokenizer to offline tokenization; causal or chunk-wise variants are needed for streaming.

Abstract

from arXiv · show

Speech codecs serve as bridges between continuous speech signals and large language models, yet face an inherent conflict between acoustic fidelity and semantic preservation. To mitigate this conflict, recent works increasingly adopt dual-tower architectures to decouple semantic and acoustic modeling with separate encoders. However, these dual-tower designs incur substantial architectural overhead. To avoid such complexity, we revisit the single-tower paradigm and propose BiMTokenizer, a low-bitrate speech codec (around 1.1 kbps) combining a bidirectional state-space backbone with Residual Spherical Leech Quantization (RSLQ). The bidirectional backbone strengthens temporal modeling, while RSLQ offers a fixed, well-separated lattice bottleneck for robust semantic and acoustic tokenization without learned-codebook collapse. Experiments show that BiMTokenizer achieves superior acoustic reconstruction and the lowest WER among low-bitrate codec baselines across both clean and noisy environments, while using less than half the parameters of recent dual-tower baselines. Furthermore, its robust semantic representations yield strong performance on downstream speech understanding tasks, confirming that a well-designed single-tower codec can preserve the semantic-acoustic balance at low bitrates. The code and model weights are available at https://github.com/ZhangXinWhut/BiMTokenizer.

1 Introduction

Speech codecs must preserve both semantic content and acoustic fidelity for Speech LLMs, but these objectives compete at low bitrates. BiMTokenizer addresses this tension with a single-tower design combining bidirectional state-space modeling and fixed-lattice quantization.

  • Speech codecs convert continuous waveforms into discrete tokens that enable autoregressive modeling and downstream speech tasks in Speech LLMs.
  • Semantic tokens preserve linguistic content but lose fine acoustic detail, whereas acoustic tokens preserve fidelity but align weakly with text-based LLMs.
  • Low-bitrate codec design is shaped by the competing objectives of semantic preservation and acoustic fidelity.
  • BiMTokenizer is a roughly 1.1 kbps single-tower codec coupling a bidirectional state-space backbone with a fixed-lattice quantizer for joint semantic-acoustic tokenization.
  • BiMTokenizer achieves superior acoustic reconstruction and the lowest WER among low-bitrate baselines in clean and noisy conditions, using fewer than half the parameters of recent dual-tower baselines.
  • Experiments on LibriSpeech and ARCH evaluate acoustic reconstruction, semantic retention, component contributions, and efficiency.

2 Related Work

Speech codec research has progressed from single-tower semantic-aware designs toward dual-tower architectures because low-bitrate semantic-acoustic balance remains limited. BiMTokenizer questions whether that shift is necessary by strengthening the shared backbone and quantization bottleneck.

  • Single-tower codecs such as SpeechTokenizer and Mimi simplify inference but have limited semantic-acoustic balance at low bitrates.
  • Recent codecs increasingly use dual towers to separate semantic and acoustic modeling, motivating a reassessment of the single-tower paradigm.
  • Bidirectional state-space models provide structured recurrent dynamics for temporal speech modeling and can replace or complement attention in speech applications.
  • Modern codecs commonly use learnable VQ or RVQ, while LFQ and BSQ remove learned codebooks but retain entropy regularization for codebook utilization.
  • FSQ avoids learned codebooks and complex entropy penalties, but its per-dimension level design is heuristic and less natural for residual coding.
  • SLQ uses a large, fixed, well-separated codebook derived from the first shell of the Leech lattice, reducing reliance on learned-codebook optimization and auxiliary constraints.

3 BiMTokenizer

BiMTokenizer is a single-tower, low-bitrate codec that combines bidirectional Mamba processing with fixed-lattice RSLQ and training-only semantic supervision. Its split quantization assigns semantic and acoustic roles within one discrete stream while avoiding learned-codebook losses.

  • 3 BiMTokenizer: BiMTokenizer combines a bidirectional Mamba encoder–decoder, an RSLQ bottleneck, and an auxiliary frozen semantic-supervision branch.The semantic branch is removed at inference, leaving the acoustic codec autonomous.
  • 3.2 Bidirectional Mamba Backbone: Bidirectional Mamba processes forward and reversed sequences, fusing their outputs to provide broader temporal context with a residual pre-norm block layout.The external bidirectional design uses two Mamba branches with independent projections and element-wise output fusion.
  • 3.3 Residual Spherical Leech Quantization: RSLQ projects features onto a normalized 24-dimensional Leech-lattice sphere and selects fixed codewords by maximum inner product.Only the projection layers and scale are learned; straight-through estimation passes gradients through nearest-codeword assignment.
  • 3.3 Residual Spherical Leech Quantization: The first RSLQ layer models semantic content, while subsequent residual layers encode speaker traits, prosody, and fine acoustic detail.With five layers at 12.5 Hz and approximately 17.58 bits per codeword, the codec operates at about 1.10 kbps.
  • 3.4 Auxiliary Semantic Supervision: Training uses reconstruction, adversarial, feature-matching, cosine semantic-distillation, and reconstruction-alignment objectives.The frozen teacher, projector, and alignment path are discarded at inference, and fixed RSLQ requires no codebook, commitment, or entropy losses.

4 Experimental Setup

The experiments train and evaluate BiMTokenizer across reconstruction and semantic-representation settings. Reconstruction is measured on LibriSpeech test-clean, while ARCH tests pooled codec representations on emotion, intent, and digit-recognition tasks.

  • 4 Experimental Setup: BiMTokenizer is trained on 960 hours of 16 kHz LibriSpeech using randomly cropped 2-second segments.The model has 253M parameters and is instantiated with Whisper-small or SenseVoice-small semantic teachers.
  • 4 Experimental Setup: Training runs for 1,000,000 steps on two NVIDIA H100 GPUs with an effective batch size of 128.Generator and discriminator optimization uses AdamW with cosine annealing after a 30k-step warmup.
  • Speech Reconstruction: Reconstruction evaluation on LibriSpeech test-clean reports intelligibility, perceptual quality, speaker similarity, and codec operating metrics.Metrics include STOI, WER, PESQ-NB/PESQ-WB, UTMOS, ViSQOL, and SIM.
  • Speech Representation: ARCH evaluates semantic representations on emotion recognition, intent classification, and digit recognition using average-pooled quantized tokens and linear classifiers.The comparison includes SSL references such as wav2vec 2.0, data2vec, HuBERT, and WavLM.

5 Experimental Results and Discussion

BiMTokenizer delivers strong acoustic reconstruction and semantic representations at 1.1 kbps, while maintaining substantially lower computational cost than dual-tower codecs. Its backbone, quantizer, and teacher choices each affect the acoustic–semantic trade-off and scaling behavior.

  • Speech Reconstruction Results: BiMTokenizer-Whisper achieves the best SIM, STOI, PESQ-NB, PESQ-WB, UTMOS, and WER among compared low-bitrate codecs at 1.1 kbps and 12.5 Hz.It reaches 3.56 PESQ-NB and 3.03 PESQ-WB, exceeding all codecs below 1.5 kbps.
  • Speech Reconstruction Results: BiMTokenizer preserves sharper harmonic bands and clearer local spectral transitions than prior low-bitrate codecs.The visual comparison complements its PESQ and UTMOS gains, with consistent gains also reported on noisier and out-of-distribution benchmarks.
  • Semantic Representation Results: Whisper-small supervision yields stronger PESQ and WER, while SenseVoice-small supervision delivers stronger semantic transfer through closer alignment with the 12.5 Hz bottleneck.The variants differ only in their frozen teacher, and the contrast suggests teacher frame rate shapes supervision interaction with the codec bottleneck.
  • Semantic Representation Results: BiMTokenizer-SenseVoice achieves the highest average ARCH accuracy, surpassing XY-Tokenizer by 1.21 absolute points while remaining single-tower.It performs best among codec systems on SLURP and AudioMNIST, whereas BiMTokenizer-Whisper performs best on RAVDESS.
  • Further Analysis: BiMTokenizer uses 253M parameters and 14.82G MACs, compared with 664M and 40.98G for DualCodec and 520M and 82.02G for XY-Tokenizer.It reduces total MACs by 63.8% and 81.9%, respectively, while achieving a total RTF of 0.007.
  • Further Analysis: BiMamba becomes more efficient as input duration grows, whereas bidirectional RoPE-based self-attention incurs steeper computation and runtime growth.Controlled backbone comparisons isolate the sequence-modeling design as the architectural difference.
  • Further Analysis: Replacing BiMamba with bidirectional self-attention raises WER from 2.55% to 4.15% and lowers PESQ from 3.56/2.99 to 3.28/2.77.Replacing RSLQ with GroupFSQ or RVQ likewise degrades all reconstruction metrics.

6 Conclusion

BiMTokenizer is a 1.1 kbps single-tower codec that combines a bidirectional state-space backbone with Residual Spherical Leech Quantization. It matches or surpasses recent dual-tower codecs with far fewer parameters, supporting single-tower redesign as a viable low-bitrate approach.

  • BiMTokenizer is a 1.1 kbps single-tower speech codec built on a bidirectional state-space backbone and Residual Spherical Leech Quantization.
  • BiMTokenizer matches or surpasses recent dual-tower codecs on reconstruction and semantic benchmarks while using far fewer parameters.
  • Careful redesign of a single-tower codec’s shared backbone and bottleneck remains an equally viable path for low-bitrate speech tokenization.

Limitations

BiMTokenizer has three stated limitations: a large fixed codebook, heterogeneous CNN–Mamba architecture, and dependence on future frames for bidirectional processing.

  • Codebook capacity: The fixed 196,560-entry RSLQ codebook enlarges autoregressive TTS prediction space and may require more data and capacity for infrequent symbols.The authors identify a capacity–predictability trade-off for future study.
  • Architecture: The current codec combines convolutional encoder/decoder modules with Mamba blocks rather than using a homogeneous architecture.A fully Mamba-based codec is proposed as a future direction.
  • Deployment: The bidirectional backbone requires future frames, limiting the codec to offline tokenization.Causal or chunk-wise variants are needed for low-latency streaming.

Ethical Considerations

The paper frames ethical considerations as primarily determined by downstream applications rather than by the representation-level codec itself. Its method uses bidirectional processing, fixed RSLQ levels, and teacher supervision during training, while codebook usage remains stable after convergence.

  • Ethical scope: Ethical risks such as privacy, bias, and misuse are primarily determined by downstream applications.The authors encourage responsible use according to ethical guidelines.
  • RSLQ organization: Five RSLQ levels divide labor between a supervised semantic level and four residual acoustic levels.The first level receives semantic supervision, while the remaining levels form the residual acoustic path.
  • Training versus inference: The semantic teacher and auxiliary alignment paths are removed at inference, leaving a single-tower encoder–RSLQ–decoder system.Teacher representations support semantic distillation and reconstruction alignment only during training.
  • Codebook behavior: All RSLQ levels remain stable after convergence, with the semantic level showing normalized entropy of about 0.81.The semantic level has lower, stable usage, while acoustic residual levels retain higher utilization and entropy.

C Training Objective Details

The training objective combines offline semantic-teacher supervision, multi-scale mel reconstruction, adversarial losses, and discriminator feature matching. The codec uses fixed RSLQ levels and reports reconstruction comparisons on LibriSpeech test-other and Seed-TTS-Eval.

  • Semantic supervision: The semantic projector maps RSLQ embeddings to the frozen teacher feature space, using upsampling for Whisper and a linear layer for SenseVoice.The projector design depends on the selected teacher.
  • Reconstruction loss: The reconstruction objective uses multi-scale mel-spectrogram losses to focus learning on perceptually relevant spectral structure.The method does not add a time-domain waveform L1 term.
  • Adversarial and feature-matching losses: Adversarial training uses multi-period and multi-scale STFT discriminators, with feature matching to stabilize training.Because RSLQ has no learnable codebook, the method adds no VQ codebook, commitment, or entropy-regularization loss.
  • Evaluation references: Reconstruction comparisons include the noisier LibriSpeech test-other set and Seed-TTS-Eval.The cited table captions identify best and second-best codec results for these comparisons.

D Evaluation Details

Evaluation measures intelligibility, acoustic quality, speaker similarity, and zero-shot speech generation across LibriSpeech and Seed-TTS-Eval. The generation setup retrains a TTS-oriented tokenizer and evaluates English and Chinese subsets.

  • Model-level comparison: LibriSpeech test-clean evaluation reports bitrate, frame rate, quantizer count, architecture status, parameters, STOI, and WER.WER transcriptions come from a HuBERT-based ASR model.
  • Acoustic evaluation: Acoustic evaluation reports PESQ-NB, PESQ-WB, UTMOS, ViSQOL, and speaker similarity.Speaker similarity is cosine similarity between original and reconstructed speech speaker embeddings.
  • Speech generation setup: The TTS-oriented BiMTokenizer is retrained on approximately 96.7K hours of English and Chinese Emilia speech.Its spherical Leech codebook is subsampled to 2,048 entries per quantization level for tractable token-level language-model training.
  • Speech generation evaluation: Zero-shot generation is evaluated on English and Chinese Seed-TTS-Eval subsets using WER, speaker similarity, and UTMOS.Inference conditions on prompt text, reference speech, and target text before decoding generated tokens into waveforms.

F Reconstruction under Noisy and Out-of-Distribution Conditions

BiMTokenizer remains robust under noisy speech and generalizes across speakers, recording conditions, content styles, and an unseen language. It also offers a controlled efficiency-analysis framework for comparing codec computation and inference cost.

  • Noisy Conditions: BiMTokenizer remains strong on noisy speech, with the Whisper-supervised variant achieving the best codec results on SIM, STOI, PESQ-NB, PESQ-WB, and WER.The SenseVoice-supervised variant achieves the highest UTMOS, but its advantage over Whisper supervision is limited to that metric.
  • Out-of-Distribution: The codec-internal ranking is preserved across English and Mandarin, indicating transfer of acoustic and phonetic regularities beyond the training language.BiMTokenizer is trained exclusively on English LibriSpeech, while Mandarin is entirely unseen during training.
  • Out-of-Distribution: BiMTokenizer-Whisper leads most reconstruction metrics on both English and Mandarin Seed-TTS subsets, while BiMTokenizer-SenseVoice leads UTMOS.Whisper leads PESQ-NB, PESQ-WB, and STOI on both subsets; SenseVoice reaches UTMOS values of 3.91 on English and 3.23 on Mandarin.
  • Conclusion: BiMTokenizer is reported as robust under difficult acoustic conditions and generalizable to out-of-distribution and cross-lingual data while retaining a single-tower architecture.The comparison with recent dual-tower codecs supports improving temporal modeling and quantization as an alternative to architectural separation.
  • Efficiency Analysis: The efficiency setup reports parameter count once per model and separately reports encoding, decoding, and summed total inference costs.Encoding converts waveforms to tokens, while decoding reconstructs waveforms from discrete tokens.
  • Efficiency Analysis: MACs measure arithmetic complexity, whereas RTF additionally reflects hardware and operator implementation, so both separate computational footprint from realized wall-clock speed.The efficiency analysis reports MACs for 1-second audio and RTF relative to audio duration.

H Component Analysis Details

Controlled component and supervision analyses identify BiMamba and RSLQ as important for reconstruction, while semantic supervision improves understanding but creates a semantic–acoustic trade-off at excessive strength.

  • Core-Design Block: The ablations hold bitrate and semantic supervision constant, attributing observed differences to the backbone or quantizer.The backbone replacement changes only BiMamba blocks, and the quantizer comparisons replace RSLQ with GroupFSQ or RVQ.
  • Core-Design Block: BiMamba with RSLQ achieves the strongest result on every reconstruction metric in the reconstruction-only block.Replacing BiMamba with self-attention mainly affects intelligibility and spectral fidelity, while replacing RSLQ causes smaller but consistent degradations across SIM, UTMOS, PESQ, and WER.
  • SenseVoice Block: 18.48 SLURP accuracy and 98.02 AudioMNIST accuracy result when SenseVoice distillation at λsem = 15 is introduced, compared with 8.58 and 81.25 without it.The teacher-specific block retains BiMamba+RSLQ and varies only semantic objectives.
  • SenseVoice Block: 2.53% WER and 98.02 AudioMNIST accuracy are obtained with reconstruction alignment enabled, improving from 2.58% and 97.79.These comparisons isolate the contribution of reconstruction alignment under λsem = 15.
  • Supervision Choices: Increasing λsem to 20 yields the highest SLURP accuracy but reduces SIM and PESQ, showing that excessive semantic supervision compromises acoustic preservation.The Whisper block shows a similar trade-off, with λsem = 20 slightly degrading UTMOS, PESQ, and WER.
  • Supervision Choices: λsem = 15 and λalign = 1 are selected for both final variants as a consistent operating point balancing semantic retention and acoustic fidelity.This choice follows the supervision-weight comparisons in the SenseVoice and Whisper blocks.
Loading 2609.00562v1…