PaperlayerView exact revision on arXiv

Source-linked AI summary

GS-Codec: A Gaussian-Splatting Bottleneck for Neural Audio Coding

Ron Aluf, Alon Canfi, Eliya Nachmani

arXiv:2610.04651v1cs.SDcs.AIcs.LGeess.ASeess.SP

TL;DR

Dominant neural audio codecs partition encoder latents through codebooks or fixed scalar grids, motivating the question of whether that partition is necessary. GS-Codec uses a Gaussian-splatting decomposition and a predictor for inference, achieving competitive or superior codec performance across several audio-quality dimensions.

  • Problem

    Dominant codec bottlenecks partition latents through learned codebooks or fixed scalar grids, and whether continuous parametric decomposition can replace that partition remains open.

    Source: §1 · p. 2

  • Method

    GS-Codec fits encoder latents as weighted sums of one-dimensional Gaussian primitives and uses a predictor to output primitive parameters in one forward pass.

    Source: §1 · p. 2

  • Results

    GS-Codec achieves competitive or superior perceptual quality, intelligibility, and speaker similarity compared with established open-source codecs across multiple bitrate regimes.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Parametric latent decompositions are a promising alternative to partition-based neural audio compression and offer a direction for fine-grained post-training rate control.

    Source: §5 · p. 10

    The evaluation covers English speech at moderate bitrates, while generalisation to other audio domains and sub-kbps coding remains future work.

    Source: F Limitations · p. 21

Abstract

from arXiv · show

Neural audio codecs compress waveforms into compact discrete tokens that underpin speech language models, real-time communication, and large-scale audio storage. Almost every dominant design, including residual vector quantization, finite scalar quantization, and single-codebook variants, follows the VQ-VAE template by partitioning the encoder latent through a learned codebook or a fixed scalar grid. We ask whether this partition is necessary. We introduce GS-Codec, a neural speech codec whose bottleneck is a parametric signal decomposition rather than a quantizer. We adapt Gaussian splatting from 3D scene reconstruction to one-dimensional latents. An inner optimization loop fits each encoder segment as a weighted sum of 1D Gaussian primitives. The decoder then reconstructs the waveform from the rendered sum. To avoid the cost of this iterative inner loop at inference time, we additionally train a lightweight GS Predictor Net that regresses the primitive parameters in a single forward pass. The encoder and decoder are trained end-to-end through the inner loop, with no quantizer anywhere in the training pipeline: the bottleneck is the decomposition itself, and scalar quantization is applied only post-training to the fitted parameters. Rather than relying on discrete codebook stages for bitrate control, our representation exposes a fine-grained rate-quality tradeoff: a single trained checkpoint supports post-training bitrate control by varying the number of primitives and the per-parameter bit depth, with no retraining required. GS-Codec matches or exceeds well-established open-source codecs such as EnCodec and DAC on speaker similarity (SIM), intelligibility (STOI), and perceptual quality (UTMOS) at comparable bitrates, while achieving comparable semantic performance (WER). Code and audio samples are available at https://ronaluf.github.io/gs-codec/

1 Introduction

Neural audio codecs provide compact tokens for audio generation and communication, but dominant bottlenecks rely on discrete partitions with limited bitrate flexibility. GS-Codec replaces that bottleneck with a Gaussian-splatting decomposition and adds a predictor for faster inference.

1 Introduction

  • Neural audio codec tokens support speech and music generation, voice conversion, dialogue, and real-time communication, with token fidelity directly bounding synthesis quality in LM-based TTS and music generation.The passages describe codecs as foundational infrastructure for multimodal audio systems.Source: §1 · p. 1
  • Dominant codecs use learned discrete codebooks, adjust bitrate in discrete stage-based steps, and can waste capacity through codebook collapse, leaving continuous parametric decomposition an open question.EnCodec and DAC are cited as near-transparent at 3–6 kbps.Source: §1 · pp. 1–2
  • GS-Codec replaces vector or scalar quantization with a weighted sum of localized Gaussian primitives, applying scalar quantization only to fitted parameters after training.A single checkpoint supports fine-grained bitrate control by changing primitive count and per-parameter bit depth without retraining.Source: §1 · p. 2
  • The GS Predictor Net approximates iterative Gaussian fitting in one forward pass, amortizing the optimization loop for faster inference.This is presented as a lightweight predictor for primitive parameters.Source: §1 · p. 2
  • GS-Codec matches or exceeds open-source codec baselines on perceptual quality and speaker similarity, while offering post-training bitrate control across LibriTTS, LJSpeech, and LibriSpeech.Bitrate is adjusted by varying primitive count and parameter bit depth without retraining.Source: §1 · p. 2

2 Related Work

Related work traces neural audio coding from codec architectures and vocoder improvements to quantizer variants, downstream applications, and evaluation metrics. GS-Codec positions its bottleneck as a parametric decomposition distinct from codebook-based rate control and per-clip neural fitting.

2 Related Work

  • SoundStream [97], Lyra V2, EnCodec [16], and DAC [42] established RVQ as a dominant bottleneck, while vocoder advances and multi-scale spectrogram losses drove fidelity gains adopted in codec pipelines.Related variants address quantizer structure, benchmarking, semantic alignment, single-codebook compression, real-time dialogue, and dynamic frame rates.Source: §2 · p. 2
  • VQ-VAE [83] [66] and Gray underpin codebook quantization, while FSQ [57] uses a fixed scalar grid; GS-Codec applies scalar quantization only after fitting continuous parameters.The passage contrasts this with bottlenecks that use straight-through gradients for discretization.Source: §2 · p. 2
  • Codec tokens support language models, self-supervised encoders, and multimodal architectures, and evaluation commonly uses PESQ, UTMOS [69], and STOI.Source: §2 · pp. 2–3
  • Per-clip SIREN fitting for audio [45] shares GS-Codec’s per-signal fitting spirit, but GS-Codec uses an analytic Gaussian decomposition with directly quantisable parameters; FlexiCodec [51] and WavTokenizer [31] instead rely on fixed codebooks for rate control.The related-work passage also identifies the SEANet backbone, Snake activation, and AdamW optimizer used in GS-Codec.Source: §2 · p. 3

3 Method

GS-Codec replaces a discrete quantizer with a one-dimensional Gaussian-splatting bottleneck: an encoder latent is fitted by Gaussian primitives and rendered for decoding. Training optimizes the encoder and decoder through this bottleneck, while inference predicts and quantizes primitive parameters for transmission.

3 Method

  • The encoder maps audio to a continuous latent, which Gaussian splatting fits as a weighted sum of one-dimensional primitives before the decoder reconstructs the waveform.The pipeline uses an iterative inner optimization loop during training.Source: §3 · p. 3, Figure 1 · p. 3

3.1.1 Architecture

  • Each primitive has a temporal center and scale plus per-channel amplitudes; iterative gradient updates fit their rendered sum to the encoder latent.Figure 2 visualizes the target, rendered approximation, and individual primitives across optimization steps.Source: §3.1.1 · p. 4, Figure 2 · p. 4

3.1.2 Loss Functions

  • Training combines latent-space commitment with waveform reconstruction, multi-scale spectrogram, adversarial, and feature-matching objectives.The commitment loss pulls the encoder output toward the splatted reconstruction.Source: §3.1 · p. 3, §3.1.2 · p. 5

3.2.1 Parameter Quantization and Token Format

  • At inference, fitted Gaussian parameters are scalar-quantized for transmission; with frozen positions, scales and amplitudes determine the bitrate, while learned positions add transmitted parameters.The parameter bit depth and number of primitives provide rate-control choices.Source: §3.2.1 · p. 5, §3.2 · p. 5

3.2.2 GS Predictor Net

  • A GS Predictor Net regresses Gaussian parameters from the latent in one forward pass, replacing the costly inner loop while keeping reconstructed quality close to the iterative fit.The predictor is trained after the encoder and decoder are frozen.Source: §3.2.2 · pp. 5–6

4 Experiments

Experiments compare GS-Codec with open-source codecs across datasets, bitrate settings, and listening tests, then examine bitrate sensitivity, position choices, and encoding speed. Results show strong reconstruction and speaker-similarity performance, while the predictor reduces encoding time relative to iterative fitting.

4.1 Experimental setup

  • Training uses a 2,339-hour mixture for the main comparison and ablations, while the LibriSpeech comparison uses a separate 54,000-hour Librilight-Large training set.Evaluation covers LibriTTS clean and noisy conditions, LJSpeech, and LibriSpeech; metrics include UTMOS, PESQ, STOI, V/UV F1, SIM, and WER.Source: §4.1 · p. 6
  • The experiments compare against EnCodec, DAC, SpeechTokenizer, HiFi-Codec, Vocos, WavTokenizer, Mimi, and XYTokenizer, with baselines drawn from published evaluations.The codec uses NG=80 primitives per 3-second segment; the iterative evaluation variant runs 300 optimization steps.Source: §4.1 · pp. 6–7

4.2.1 Comparison with open-source codecs

  • At NG=90 and 6-bit quantization, GS-Codec surpasses comparable-bitrate baselines on UTMOS, PESQ, and STOI across three test sets, while lower-rate performance degrades gracefully.At 2.84 kbps, it still leads multi-codebook baselines at 3 kbps on UTMOS; WavTokenizer is stronger below 1 kbps.Source: §4.2.1 · p. 7, Table 1 · p. 8
  • On LibriSpeech, GS-Codec leads on UTMOS and SIM while DAC leads on NB-PESQ; same-backbone tests find higher UTMOS and SIM with comparable PESQ-NB, STOI, and WER.Table 2 reports 4.01 UTMOS and 0.94 SIM for GS-Codec, while DAC has a 3.77 versus 3.71 NB-PESQ advantage.Source: §4.2.1 · p. 7, Table 2 · p. 9, Table 3 · p. 9
  • At approximately 6 kbps, blinded listening tests favor GS on mean MUSHRA and MOS gaps to natural speech; at approximately 3 kbps, MOS gaps are smaller at similar MUSHRA scores.The comparisons use FSQ and RVQ, with DAC and EnCodec as external reference points.Source: §4.2.1 · p. 7

4.2.2 Bitrate sweep

  • At fixed 5-bit depth, metrics improve steeply through NG=80 and then saturate as NG increases from 50 to 140.The sweep spans 2.75–7.70 kbps.Source: §4.2.2 · p. 7, Figure 3 · p. 7

4.2.3 Ablations

  • Uniform-grid primitive positions avoid transmitting learned positions, whose additional bitrate does not yield large perceptual gains at sufficient primitive counts.Figure 5 compares absolute UTMOS and PESQ, plus learned-minus-frozen deltas, across NG at fixed 6-bit quantization.Source: §4.2.3 · p. 7

4.3 Encoding time and throughput

  • The GS Predictor Net reduces encoder time to 7.9 ms per segment (RTF 0.003), versus 171 ms (RTF 0.057) for iterative GS, though it remains slower than DAC and EnCodec.Timing excludes the decoder; the predictor measurement uses frozen centers.Source: §4.3 · p. 9, Table 4 · p. 9

5 Conclusion

The conclusion presents GS-Codec as a parametric Gaussian-splatting alternative to conventional quantization bottlenecks. Experiments report competitive or superior codec quality and a single-pass predictor, while bitrate sweeps show diminishing gains beyond selected settings.

5 Conclusion

  • GS-Codec replaces conventional quantization bottlenecks with a parametric one-dimensional Gaussian-splatting decomposition of encoder latents.The representation uses a weighted sum of Gaussian primitives rather than RVQ or FSQ.Source: §5 · p. 9
  • GS-Codec achieves competitive or superior perceptual quality, intelligibility, and speaker similarity, while the Predictor Net amortizes iterative fitting into one forward pass with reconstruction quality close to the full fit.The conclusion describes quality as maintained close to the optimization-based approach.Source: §5 · p. 9
  • For NG=60 and NG=80, metrics saturate past 6 bits, and NG=80 outperforms NG=60 at every tested bit depth.Figure 5 also contrasts frozen and learned primitive positions using UTMOS and PESQ across NG, including their metric deltas.Source: Figure 4 · p. 10, Figure 5 · p. 10
  • The results suggest parametric latent decompositions are a promising alternative to partition-based compression and support fine-grained post-training rate control.The conclusion frames this as a direction for neural codecs.Source: §5 · p. 10

A Training and evaluation details

The appendix specifies the codec architecture, Gaussian bottleneck settings, training schedule, evaluation configuration, and Predictor Net training procedure.

A Training and evaluation details

  • The 24 kHz SEANet encoder-decoder uses 32 latent channels and 64 base channels, producing 225 latent frames per 3-second segment at 75 Hz.It uses Snake activations, strided ratios [8] [5] [4], two LSTM layers, and non-causal bidirectional convolutions.Source: A Training and evaluation details · p. 18
  • The Gaussian bottleneck uses 80 primitives, 100 inner training steps, and 300 evaluation steps, with centers fixed to a uniform grid and scales shared across channels.Evaluation uses a cosine learning-rate schedule and explicit bounds for scale and amplitude parameters.Source: A Training and evaluation details · p. 18
  • Training combines time-domain L1, multi-scale spectrogram, adversarial, and feature-matching losses; SI-SNR is disabled, and Adam uses a 10^-4 learning rate with EMA.The schedule runs 200 epochs of 2,000 steps on one RTX PRO 6000 GPU, for approximately 144 hours.Source: A Training and evaluation details · p. 18
  • Evaluation specifies the WER model, duration-filtered LibriSpeech subset, PESQ bandwidth variants, UTMOS model, and WavLM-Large speaker-similarity measure.The Predictor Net is trained after the codec with encoder and decoder frozen, using only a splat-reconstruction penalty.Source: A Training and evaluation details · p. 18

B Per-parameter quantisation ranges

The appendix reports the per-parameter quantization ranges used for the headline checkpoint.

B Per-parameter quantisation ranges

C Controlled evaluation

Controlled evaluations compare GS with quantized and continuous-autoencoder bottlenecks, test representation choices, and examine subgroup and training-cost protocols. The reported results favor the Gaussian decomposition over the continuous-autoencoder control and show that removing the decomposition has a larger effect on latent NMSE than changing the atom family.

Unified evaluation setup

  • At 5.950 kbps, GS has the lowest temporal rate but 1.93× RVQ’s token rate and 0.48× FSQ’s token rate.The unified evaluation uses a pinned 1,237-utterance LibriSpeech test-clean set; GS uses 102 primitives with 5-bit scales and amplitudes and 10-bit centers.Source: C Controlled evaluation · p. 19, Table 6 · p. 19

Subgroup analysis

  • The subgroup protocol compares GS Predictor with same-backbone FSQ and RVQ across extreme terciles of five speech characteristics.Each tercile contains 412 of the 1,237 utterances; the cited passages provide the analysis design but no subgroup outcomes.Source: C Controlled evaluation · p. 19, Table 7 · p. 19

Representation ablation

  • Removing the decomposition roughly doubles latent NMSE, while changing the atom family changes it by at most 17%.The representation ablation holds payloads within 0.85% of 17,850 bits and uses plain post-training quantization.Source: C Controlled evaluation · p. 19, Table 8 · p. 20

Continuous-autoencoder control

  • At its trained count, the continuous autoencoder trails GS by 0.066 UTMOS with fewer bits; away from that count, it trails by 0.85–0.88 UTMOS at matched bits.The comparison uses the same backbone, with the autoencoder trained at 102 frames and evaluated at and away from that count.Source: C Controlled evaluation · p. 19, Table 9 · p. 20

Training cost

  • Training-cost measurements use a single L40S with bf16 and batch size 16; the batched inner fit’s history is not traversed by the outer backward pass.The cited table caption identifies training cost per update, but the supplied passages do not give the measured timings.Source: C Controlled evaluation · p. 20, Table 10 · p. 20

D Human listening evaluation

The human listening evaluation compares GS configurations with FSQ and RVQ using MOS and MUSHRA ratings. At about 6 kbps, GS has higher mean MUSHRA and smaller mean MOS gaps; at about 3 kbps, it has smaller MOS gaps at similar MUSHRA.

Evaluation protocol

  • Ten LibriSpeech test-clean utterances, one per speaker, were selected randomly; listeners rated MOS on a 1–5 scale and MUSHRA on a blinded 0–100 interface.MOS gaps are natural-speech minus system ratings, paired within listener; intervals use participant-level bootstrap.Source: D Human listening evaluation · p. 20, Table 11 · p. 21

Listening scores

  • At ≈6 kbps, GS configurations obtain higher mean MUSHRA and smaller mean MOS gaps than FSQ and RVQ; at ≈3 kbps, they obtain smaller mean MOS gaps at similar MUSHRA.The Predictor’s MOS gap is smaller at ≈3 than at ≈6 kbps, with overlapping intervals; its MUSHRA ordering is as expected.Source: D Human listening evaluation · p. 20, Table 11 · p. 21

E Fixed-budget allocation

The fixed-budget evaluation compares allocations between primitive count and parameter precision with frozen positions. At 5.28 kbps, the higher-capacity, lower-precision allocation outperforms the alternatives on three reported quality and similarity metrics.

Fixed-budget allocation

  • At 5.28 kbps, 120g/4b beats 160g/3b and 80g/6b on PESQ-NB, UTMOS, and SIM.At NG=60, increasing precision from 5 to 10 bits moves UTMOS only from 2.845 to 2.897; the passage concludes that sufficient primitive capacity comes first, with remaining budget for precision.Source: E Fixed-budget allocation · p. 21, Table 12 · p. 21

F Limitations

The paper reports limits in encoding speed, low-bitrate perceptual quality, evaluated audio scope, and streaming applicability. It also notes privacy and misuse concerns for reconstructed audio and downstream speech synthesis.

F Limitations

  • Encoding remains slower than EnCodec and DAC even with the predictor; experiments cover English speech at moderate bitrates, while other audio domains and sub-kbps coding remain future work.The codec processes complete 3-second segments with non-causal components, so GPU execution time does not represent streaming latency.Source: F Limitations · p. 21

G Broader impact

  • Reconstructed audio can retain speaker identity and private information, so use should respect data consent and licenses.Downstream speech-synthesis applications share the potential misuse risks of existing audio representations.Source: G Broader impact · p. 22

Related papers

Loading 2610.04651v1…