PaperlayerView exact revision on arXiv

Source-linked AI summary

PACodec: A Low-bitrate Neural Speech Codec with Parallel Additive Vector Quantization

Fei Liu, Yang Ai, Xiao-Hang Jiang, Zhen-Hua Ling

arXiv:2609.03363v1cs.SD

TL;DR

RVQ’s sequentially dependent quantizers make further bitrate reduction difficult and constrain its use in speech disentanglement, while FSQ requires an excessively large codebook for competitive performance. PACodec uses parallel additive VQs to quantize local aspects of shared global features and add their outputs. It reduces bitrate by about 30% versus baselines at the same decoding quality, and its branches capture complementary speech information.

  • Problem

    RVQ’s sequentially dependent VQs make further bitrate reduction difficult and restrict its applicability to disentanglement tasks, while FSQ requires an excessively large codebook for competitive performance.

    Source: §I · p. 1

  • Method

    PACodec’s PAVQ independently quantizes local aspects of shared global encoded features in parallel and adds branch outputs into a global quantized result.

    Source: §I · pp. 1–2

  • Results

    About 30% bitrate reduction versus baselines at the same decoding quality, with 6.7M parameters, the smallest count among the compared codecs.

    Source: §I · p. 2, C. Speech Coding Performance Comparison · p. 4

  • Takeaways & Limitations

    Different PAVQ branches capture complementary speech information, including content, timbre, and acoustic details, indicating disentanglement potential for tasks such as voice conversion.

    Source: D. Disentanglement Potential Analysis · p. 5, §IV · p. 5

Abstract

from arXiv · show

This paper proposes PACodec, a novel low-bitrate neural speech codec based on parallel additive vector quantization (PAVQ). Unlike the mainstream residual vector quantization (RVQ) used in most neural speech codecs, where vector quantizers (VQs) are sequentially dependent, the PAVQ strategy adopted in PACodec aggregates parallel quantization results to optimize bitrate usage. Specifically, the PAVQ adopts a "global-local-global" (GLG) design: the global encoded features are quantized in parallel by multiple independent VQs, each attending to a local component of the representation, and their outputs are aggregated through addition to yield the final global quantization result for decoding. Experimental results show that PACodec, as each VQ focuses only on local information, supports smaller codebooks and reduces bitrate by 30% compared with baselines at the same decoding quality, with only minor model complexity. Further analysis shows that, owing to the GLG framework of PAVQ, the proposed PACodec is disentanglement-friendly, and each independent VQ captures different aspects of speech, e.g., content, timbre, and acoustic details, suggesting potential for application to downstream tasks such as voice conversion.

I. INTRODUCTION

PACodec uses parallel additive vector quantization to address bitrate and disentanglement limits in existing quantization approaches. It achieves about 30% bitrate reduction versus baselines while preserving speech quality, and its branches capture complementary speech information.

I. INTRODUCTION

  • Lower bitrates support more efficient storage and transmission while preserving intelligibility and naturalness, especially in bandwidth-constrained applications.Source: §I · p. 1
  • Neural speech codecs deliver high decoded quality at reduced bitrates, while spectral-feature approaches such as MDCTCodec streamline computation relative to dual-branch APCodec.Direct waveform modeling requires multiple downsampling and upsampling operations, increasing computational complexity.Source: §I · p. 1
  • RVQ’s sequentially dependent VQs make further bitrate reduction difficult and restrict disentanglement applications, while FSQ requires an excessively large codebook for competitive performance.Grouped RVQ adds parallelism but retains RVQ and makes branch roles difficult to interpret.Source: §I · p. 1
  • PACodec’s PAVQ uses a global–local–global design: independent branches quantize the same global features locally, then add their outputs into a global quantized result.The method is trained with PAVQ quantization loss and speech reconstruction loss.Source: §I · pp. 1–2
  • About 30% bitrate reduction versus baselines across 16 kHz and 48 kHz datasets, with speech quality preserved.Each VQ focuses on local information, supporting codebooks as small as 128.Source: §I · p. 2

II. PROPOSED METHOD

PACodec encodes speech into features, quantizes them with PAVQ, and decodes them to reconstruct the waveform. Its spectral encoder and decoder use mirrored structures built around MDCT features.

A. Overview

  • The overview pipeline comprises a spectral encoder, PAVQ quantization, and a spectral decoder that reconstructs the waveform.Training combines PAVQ quantization loss to minimize quantization errors with reconstruction loss to narrow the gap between decoded and raw speech.Source: A. Overview · p. 2

B. Spectral Encoder & Decoder

  • The encoder transforms speech into MDCT spectral features, processes them through 1D convolutions and ConvNeXt v2 networks, then downsamples them into encoded features.Source: B. Spectral Encoder & Decoder · p. 2
  • The decoder mirrors the encoder, replacing downsampling convolutions with transposed convolutions and using inverse MDCT to reconstruct the waveform.The downsampling and upsampling ratio depends jointly on the MDCT frame shift and convolution strides.Source: B. Spectral Encoder & Decoder · p. 2

C. Parallel Additive Vector Quantization

PAVQ applies independent VQs to the same global encoded features, each quantizing a local speech aspect, and adds their outputs to form global quantized features. This avoids RVQ residual structures and channel-wise grouping while supporting smaller codebooks.

C. Parallel Additive Vector Quantization

  • PAVQ consists of N parallel VQs, each quantizing a different local aspect of the same global encoded features.The localized scheme facilitates smaller codebooks for bitrate reduction.Source: C. Parallel Additive Vector Quantization · p. 2
  • Each VQ selects a codebook vector for each encoded feature using the minimum Euclidean distance principle.Each branch has its own trainable codebook.Source: C. Parallel Additive Vector Quantization · p. 2
  • The local quantized features from all VQs are added to form the global quantized feature.The PAVQ tokens comprise one token sequence per VQ.Source: C. Parallel Additive Vector Quantization · p. 3
  • Unlike RVQ and GRVQ, PAVQ avoids residual structures and channel-wise grouping, yielding a simpler design and clearer interpretation of each VQ.Its branches capture relatively independent local information, eliminating the need for large codebooks and thereby reducing bitrate.Source: C. Parallel Additive Vector Quantization · p. 3

D. Training Criteria

PACodec is jointly trained to reduce quantization error and improve reconstruction. Its training objective combines a branchwise quantization loss with adversarial and spectral reconstruction terms.

D. Training Criteria

  • The PAVQ loss sums mean squared errors between each branch’s input and output to optimize codebooks and reduce quantization error.The loss uses the Frobenius norm.Source: D. Training Criteria · p. 3
  • The reconstruction loss measures the difference between decoded and raw speech using adversarial and spectral terms.The two losses are combined to train PACodec jointly at quantization and reconstruction levels.Source: D. Training Criteria · p. 3
  • The adversarial term uses a multi-resolution MDCT-based discriminator to judge raw and decoded speech spectra under different MDCT configurations.Source: D. Training Criteria · p. 3
  • The spectral term measures differences in MDCT spectra with MSE and in mel-spectrograms with MSE and MAE.The loss is intended to encourage perceptual alignment between decoded and raw speech.Source: D. Training Criteria · p. 3

A. Experimental Setting

Experiments use LibriTTS at 16 kHz and VCTK at 48 kHz, with unseen speakers reserved for testing. PACodec uses four 128-entry VQs, and the LibriTTS listening-test comparison pairs its 1.4 kbps rate with 2 kbps baselines.

A. Experimental Setting

  • LibriTTS provides about 585 hours of 16 kHz speech, with 354,780 training utterances and 4,837 test utterances from unseen speakers.The training and test sets contain 2,311 and 39 speakers, respectively.Source: A. Experimental Setting · p. 3
  • VCTK provides about 43 hours of 48 kHz speech, with 40,936 training utterances and 2,937 test utterances from unseen speakers.The training and test sets contain 100 and 8 speakers, respectively.Source: A. Experimental Setting · p. 3
  • PACodec uses four parallel VQs with codebook size 128, and its encoder and decoder each contain eight 1D ConvNeXt v2 blocks.The encoding and decoding stages use a downsampling and upsampling factor of 320; training runs for 500 epochs with AdamW.Source: A. Experimental Setting · p. 3
  • The ABX test compares average preference scores for PACodec at 1.4 kbps against 2 kbps baselines on 16 kHz LibriTTS.The caption defines N/P as “no preference” and p as the paired t-test p-value.Source: Figure 2 · p. 4

B. Baselines

PACodec is compared with seven neural speech codecs across bitrate settings on LibriTTS and VCTK. Baselines are retrained using their official implementations, with some settings excluded because they are unavailable or unsupported.

B. Baselines

  • The comparison includes Encodec, AudioDec, DAC, HiFi-Codec [16], APCodec, MDCTCodec, and SQCodec [20].Source: B. Baselines · p. 4
  • On LibriTTS, PACodec at 1.4 kbps is compared with baselines at 1.5 and 2 kbps; on VCTK, its 4.2 kbps rate is compared with 4.5 and 6 kbps baselines.Source: B. Baselines · p. 4
  • SQCodec is included only at 1.5 kbps on LibriTTS because other settings are unavailable, while HiFi-Codec lacks the 1.5 and 4.5 kbps settings.All baselines were retrained with their official implementations.Source: B. Baselines · p. 4

C. Speech Coding Performance Comparison

PACodec maintains comparable quality at lower bitrates on LibriTTS and delivers equal or better quality than higher-rate baselines on VCTK. It also has the fewest parameters among the compared codecs.

C. Speech Coding Performance Comparison

D. Disentanglement Potential Analysis

The analysis uses branch ablations on VCTK to examine which speech attributes each VQ captures. The results associate VQ1 with text content, VQ4 with pitch and timbre, and VQ2–VQ3 with acoustic detail.

D. Disentanglement Potential Analysis

  • The ablations assess content, timbre, and acoustic detail using WER, CER, STOI, F0-RMSE, speaker similarity, and MCD.Removing one branch and reconstructing from the others exploits PAVQ addition, which facilitates partial speech reconstruction for analysis.Source: D. Disentanglement Potential Analysis · p. 4
  • Removing VQ1 substantially degraded all metrics, especially WER, CER, and STOI, indicating a major role in text content and speech intelligibility.Its absence also affected other metrics.Source: D. Disentanglement Potential Analysis · p. 4
  • Removing VQ4 severely worsened timbre metrics, especially F0-RMSE, while content metrics declined only slightly relative to removing VQ1.The analysis associates VQ4 mainly with pitch- and timbre-related information.Source: D. Disentanglement Potential Analysis · p. 4
  • Removing VQ2 or VQ3 caused moderate metric degradation without severe declines, and these branches mainly capture acoustic detail, with VQ2 capturing more than VQ3.The paper concludes that the branches capture complementary local information and identifies voice conversion as a potential downstream application.Source: D. Disentanglement Potential Analysis · p. 5

E. VQ Number Sensitivity Analysis

PACodec reports about 30% bitrate reduction versus baselines at the same decoding quality, with minor model complexity. A VCTK experiment also finds it relatively insensitive to reducing the number of PAVQ branches from four to three.

E. VQ Number Sensitivity Analysis

  • Reducing PAVQ branches from four to three lowers bitrate from 4.2 kbps to 3.15 kbps on VCTK, with results indicating relative insensitivity to branch count.The paper describes this as robustness under varying configurations and potential for further bitrate reduction without significant performance degradation.Source: E. VQ Number Sensitivity Analysis · p. 5

IV. CONCLUSION

  • About 30% bitrate reduction versus baselines at the same decoding quality, with minor model complexity, is reported for PACodec.The conclusion attributes smaller codebooks to branches focusing on local information.Source: §IV · p. 5
Loading 2609.03363v1…