PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions

Joonyong Park, Shinnosuke Takamichi, David M. Chan, Shunsuke Kando, Yuki Saito, Hiroshi Saruwatari

arXiv:2608.31037v1cs.CLcs.SD

TL;DR

The paper asks how language-like statistical regularities in neural audio codec tokens vary across architectures, corpora, and acoustic conditions. It analyzes 13 codecs with matched samples, family-conditional n-gram orders, and fit-validity safeguards, finding that corpus identity matters little while architecture and acoustic condition dominate in metric-dependent ways. The results support architecture-conditioned conventions for analyzing codec-token statistics and identify condition-specific degradation signatures and a DEMAND-noise quality association.

  • Problem

    Prior evidence on language-like NAC token statistics was limited in architecture, corpus, and acoustic-condition coverage, leaving robustness and generalization insufficiently characterized.

  • Method

    The paper analyzes 13 NACs across three quantizer meta-categories, three corpora, and three acoustic conditions using matched token samples, family-conditional n-gram orders, fit safeguards, and variance decomposition.

  • Results

    Corpus identity explains little distributional variance, whereas acoustic condition and quantizer architecture dominate metric-dependent patterns; common-order unigram JSD tracks perceived quality most clearly under DEMAND noise.

  • Takeaways & Limitations

    Language-statistical analysis of NAC tokens should use architecture-conditioned conventions rather than a universal n-gram target, while interpreting degradation signatures by codec family and noise condition.

  • Takeaways & Limitations

    Cross-codec comparisons use family-conditional n-gram orders, so absolute parameter values are not directly comparable across families.

Abstract

from arXiv · show

Neural audio codecs (NACs) convert speech into discrete token sequences, and prior work has reported that these sequences follow language-like statistical laws. This paper analyzes the token statistics of 13 NACs spanning multi-codebook residual vector quantization (RVQ), single-codebook VQ, and non-VQ designs, evaluated on three corpora under clean, white-noise, and real-world DEMAND-noise conditions. Zipf and Heaps parameters, unigram entropy, codebook occupancy, and Jensen-Shannon divergence (JSD) are estimated from matched token samples with explicit fit-validity safeguards and family-conditional $n$-gram orders. Corpus identity explains little variance in any metric, whereas acoustic condition and quantizer meta-category dominate in a metric-dependent way, and unigram entropy is the metric most strongly associated with meta-category. Clean-to-noise JSD computed at a common unigram order is associated with mel-cepstral distortion most clearly under DEMAND noise. The collapse and explosion degradation signatures previously reported for RVQ codecs concentrate in RVQ cells under white and DEMAND noise, respectively; explosion also occurs in non-VQ codecs, and single-codebook VQ codecs shift in occupancy and distribution shape without either signature. These results provide architecture-conditioned conventions for applying language-statistical analysis to NAC tokens.

I. INTRODUCTION

Neural audio codecs convert speech into discrete tokens that support language-modeling approaches, but the statistical regularities and robustness of those tokens remain incompletely characterized. This study expands analysis across codec architectures, corpora, and acoustic conditions.

  • NACs convert continuous speech waveforms into discrete token sequences through learned codebooks, enabling sequence-modeling methods developed for natural language.
  • The study asks whether NAC tokens exhibit language-like regularities and whether those regularities reveal codec quality and robustness.
  • Prior work found Zipf-like behavior in NAC tokens, but examined only clean speech, one corpus, and a small set of RVQ codecs.Its links to resynthesis quality were also based on limited evidence.
  • The paper evaluates 13 NACs spanning multi-codebook RVQ, single-codebook VQ, and non-VQ designs across three corpora and clean, white-noise, and DEMAND-noise conditions.
  • Corpus identity contributes little, while acoustic condition and quantizer architecture shape token statistics in metric-dependent and family-specific ways.The reported degradation signatures concentrate differently across architecture families and noise conditions.

B. Statistical Laws of Language

Zipf’s and Heaps’ laws motivate statistical analysis of speech tokens, while prior work leaves NAC behavior under controlled within-speech variation and noisy conditions insufficiently characterized. The paper addresses this gap with a crossed, standardized analysis pipeline.

  • Statistical laws originally studied in language have also been reported for visual codec tokens and linked to downstream task behavior.
  • Prior speech-token studies examined interpretability, redundancy, and cross-domain distributions, including SSL-based semantic tokens and NAC tokens.
  • Earlier NAC analysis found Zipf and Heaps behavior but was restricted to one clean corpus, with modest condition-level correlations and unresolved reliability and generalization questions.
  • Noise robustness matters because real-world audio includes environmental degradation, while existing evaluation frameworks focus predominantly on clean inputs.
  • The present design evaluates 13 codecs across three corpora and clean, white-noise, and DEMAND-noise conditions using standardized extraction and family-conditional processing.The pipeline includes deduplication, n-gram mapping, chunk-based estimation, and variance decomposition.

B. Statistical Metrics

The paper formalizes language-statistical metrics for codec tokens, including Zipf and Heaps parameters, entropy and redundancy, with explicit estimation conventions and fit safeguards.

  • 1) Zipf’s Law: Zipf’s law models the frequency of the r-th ranked element as a power law f(r) ∝ r^-s.
  • 1) Zipf’s Law: The paper reports the equivalent frequency-distribution exponent α = 1 + 1/s, with natural-language word-level reference α ≈2.
  • 1) Zipf’s Law: Zipf exponent α is estimated by maximum likelihood over type frequencies, using xmin and the number of qualifying types n.Fit quality is assessed with KS distance; estimates are refit when the default bound truncates α at 3.
  • Fits are treated as undefined when the type–token ratio V/N exceeds 0.9 because the observed inventory is nearly unique and the frequency profile is degenerate.
  • 2) Heaps’ Law: Heaps’ law models vocabulary size as V(N) = K · N^β, where K reflects initial vocabulary introduction and β controls continued type growth.Natural-language word-level references typically have β ≈0.4–0.6.
  • 3) Shannon Entropy and Redundancy: Shannon entropy measures average information per token from token probabilities, while redundancy measures the gap from maximum entropy over the observed vocabulary.Token perplexity 2^H re-expresses entropy as an effective number of frequently used token types.

4) JSD:

The analysis uses Jensen-Shannon divergence to compare clean and degraded token distributions, alongside several resynthesis-quality metrics.

  • JSD compares empirical n-gram frequency distributions between clean reference P and degraded distribution Q.It is nonparametric rather than based on fitted α or β parameters.
  • JSD is symmetric, bounded in [0, 1], and defined even for non-overlapping supports.The mixture distribution is used with Kullback–Leibler divergences in the definition.
  • Resynthesized speech is evaluated after decoding complete token sequences and resampling the output to 16 kHz.Metrics are computed on the resynthesized waveform against the clean reference where applicable.
  • Quality evaluation includes dCER, UER, UTMOS, MCD, and F0 error.UER is defined only under noisy conditions, while UTMOS estimates perceived naturalness and MCD measures mel-cepstral distortion.

D. Analysis Framework

The analysis framework combines chunk-based reliability checks, occupancy measurements, and variance decomposition across codec architecture, corpus, and acoustic condition.

  • Zipf and Heaps parameters are estimated on non-overlapping chunks from 5K to 500K tokens against matched 500K-token references.Deviations from the largest reference quantify finite-sample stability.
  • Codebook occupancy measures the number of unique observed tokens normalized by an estimated codebook size.For codecs without a nominal vocabulary size, the denominator uses the largest observed token index plus one.
  • The design crosses 13 NACs with three corpora and three acoustic conditions, yielding 117 codec–corpus–condition token streams.The protocol includes clean, white-noise, and DEMAND-noise conditions.
  • The pipeline applies consecutive deduplication and family-conditional n-gram orders before chunk-based estimation.The accompanying variance analysis uses codec quantizer category, corpus, and noise condition as factors.
  • Reported η2 values are descriptive variance partitions over the cell grid, with a codec-random-intercept mixed model as a robustness check.Cells contributed by each codec are not treated as independent samples.

IV. EXPERIMENTAL SETUP

The experiment evaluates 13 codecs from three quantizer meta-categories on matched speech corpora under clean and two 0 dB noise conditions.

  • All 13 codecs are evaluated in default released configurations across 117 codec–corpus–condition cells.The setup avoids manual bandwidth adjustment.
  • The codec set spans multi-codebook RVQ, single-codebook VQ, and non-VQ alternatives.The categories are defined by quantizer topology.
  • The three corpora cover single-speaker monolingual, multi-speaker, and multilingual speech variation.They are approximately matched at ∼10 hours, and all codecs use the full corpus set.
  • Each corpus is tested clean, with Gaussian white noise at 0 dB SNR, and with DEMAND real-world noise at 0 dB SNR.DEMAND conditions pool five environments, random channels, and random start offsets using identical noisy waveforms across codecs.

A. Finite-Sample Stability

The study establishes finite-sample references, selects n-gram orders under validity safeguards, and validates extraction choices before downstream comparisons.

  • A. Finite-Sample Stability: 114 of 117 cells have matched 500K-token references; three Mimi–TrIJEK cells are excluded because their streams are shorter.Heaps intercept K uses absolute log-ratio deviations because its scale varies across codecs.
  • A. Finite-Sample Stability: At 200K tokens, KS median deviation falls from 0.55 to 0.07 reference spreads, while β, α, and K remain at 0.17, 0.17, and 0.19.KS is the only parameter reaching the 0.1 guide; α is least stable at 5K tokens with deviation 1.06.
  • B. Selection and Sensitivity of n-gram Order: Most multi-codebook RVQ codecs favor n = 3 or 4, whereas single-codebook and large-vocabulary codecs favor lower orders.Higher-order inventories can approach V ≈ N, making fits undefined.
  • B. Selection and Sensitivity of n-gram Order: The adopted n-gram order matches the corpus-mean KS optimum for eight codecs, differs by one for four, and by two for Mimi.Cross-family differences under codec-specific orders are treated as descriptive because valid fits become sparse at common higher orders.
  • 1) RVQ Level: For RVQ codecs, the first residual-quantizer level has lower KS distance than later levels in the five codecs with level-wise results.The first level is therefore retained for the full codec set.
  • 2) Deduplication: Consecutive deduplication increases estimated α and β with Cohen’s d ≈0.5 but does not significantly change KS or K.This convention is applied before fitting the distributional models.

VI. CROSS-CORPUS AND CROSS-ARCHITECTURE ANALYSIS

Clean-speech token statistics vary mainly across codec architectures rather than corpora, while noise shifts depend on both the metric and acoustic condition.

  • RVQ-family averages remain similar across LJSpeech (α = 2.32, β = 0.84), VoxCeleb (α = 2.44, β = 0.91), and TrIJEK (α = 2.42, β = 0.91).
  • Multi-codebook RVQ codecs span α ≈2.0–2.7, while SQCodec and FocalCodec sit near α ≈2.8–3.0 and S3Tokenizer at 2.41.
  • On clean speech, entropy is positively associated with vocabulary scale, while every codec uses its observed inventory close to uniformly and the single-codebook VQ group and FocalCodec are lowest.
  • White noise removes about 1–4.4 bits of entropy for most codecs and raises grid-mean redundancy from 0.08 to 0.27, whereas DEMAND noise changes entropy comparatively little.
  • The median absolute clean-to-0 dB shift in α is 0.35 under white noise and 0.23 under DEMAND noise, with affected cells shifting by up to 1.91.
  • White noise depresses α most for single-codebook VQ codecs, with ∆α = −1.02 versus −0.36 for RVQ and 0.00 for non-VQ cells.
  • Cross-noise robustness rankings are metric-dependent: UER rankings agree strongly (ρ = 0.81), occupancy shifts moderately (ρ = 0.65), and α shifts not at all (ρ = −0.04).

D. Variance Decomposition

Variance decomposition shows that acoustic condition and quantizer meta-category dominate different token statistics, while corpus identity contributes little variation in the evaluated grid.

  • Acoustic condition is the largest main effect for α, β, and redundancy R, while meta-category leads for KS, Heaps intercept K, and unigram entropy H.
  • The meta×condition interaction explains appreciable variance for α, KS, and R, indicating family-specific responses to acoustic condition, while residual variance remains large.
  • After including codec random intercepts, the joint noise effect and meta×condition interaction remain significant for every metric (Wald p ≤0.001).
  • Corpus identity explains little variation across the three-corpus grid, with corpus terms reaching significance only for β and remaining below the noise terms.
  • Clean-speech dCER is at most 0.06 for every codec except WavTokenizer, which reaches 0.21 despite MCD and F0 errors within the remaining codecs’ range.
  • XCodec2 and BigCodec exceed reference UTMOS on TrIJEK and VoxCeleb by up to 0.5, while increasing reference-based spectral distance.

B. JSD and Quality under Noise

The paper evaluates clean-to-noise token-distribution shifts with common-order JSD and finds their association with quality is clearest under DEMAND noise.

  • JSD compares equal-sized clean and noisy token samples at the common unigram order n = 1, avoiding saturation and invalidity at most higher orders.
  • Under DEMAND noise, larger JSD is associated with lower UTMOS and larger median F0 error, while the MCD association is weak after level correction.
  • The DEMAND-noise UTMOS association persists after removing two non-VQ codecs, with r = −0.68 and p = 0.029.
  • Under white noise, no association with audio-based quality metrics survives both correlation statistics; dCER associations remain weak throughout.
  • The weak DEMAND-noise JSD–UER association partly reflects uniformly high token degradation, with UER cell means spanning 0.66–1.00.
  • The sequence analysis asks whether noise produces different degradation patterns across quantizer architectures using repetition rate and transition entropy.

B. Shifts by Quantizer Architecture

Noise produces architecture-specific sequence degradation: RVQ codecs show collapse under white noise and explosion under DEMAND, while single-codebook VQ codecs deform without either signature.

  • RVQ codecs cluster in the collapse quadrant under white noise and the explosion quadrant under DEMAND noise.
  • Single-codebook VQ codecs lose 33–75 percentage points of occupancy per cell under white noise while remaining near zero repetition-rate shift, so they satisfy neither signature.
  • Non-VQ codecs show no collapse signature; S3Tokenizer crosses the explosion deadband in four of six cells, while FocalCodec and SQCodec remain neutral.
  • Of 15 collapse instances, 14 occur under RVQ–white, while explosion concentrates under RVQ–DEMAND and four additional explosion cells belong to S3Tokenizer.
  • Cross-codec comparisons should be conducted at fixed meta-category, n-gram order, and, where possible, vocabulary scale because these factors confound between-family contrasts.
  • The interpretation of α near the natural-language reference value should be family-conditional rather than treated as a universal target.

B. JSD as a Complementary Distributional Diagnostic

JSD provides a decoder-free distribution-level diagnostic that captures token-frequency redistribution and is most clearly associated with acoustic quality under real-world DEMAND noise. Its use is part of an architecture-conditioned analysis protocol with matched samples and explicit reliability safeguards.

  • JSD shows the clearest association with acoustic quality under real-world DEMAND noise.It tracks UTMOS and median F0 error in the expected directions.
  • JSD detects token-frequency redistribution that can preserve overall power-law shape while changing individual token frequencies.Unlike α or KS, it responds to probability-mass redistribution across the full token frequency distribution.
  • JSD is decoder-free and can evaluate nondecodable tokenizers such as S3Tokenizer.It uses token-level counts rather than waveform reconstruction.
  • Cross-codec JSD requires a matched low n-gram order because the measure saturates when compared samples share little n-gram support.This is one of the paper’s scope conditions for interpreting JSD across codecs.
  • The broader protocol combines family-conditional n-gram orders, fit-validity safeguards, matched-sample chunk assessment, and variance decomposition.These conventions make language-statistical analysis family-conditional rather than a universal target.
Loading 2608.31037v1…