Source-linked AI summary
Multi-Rate Bandwidth Extension by Token Completion in Neural Audio Codecs
Benoît Ginies, Olivier Fercoq, Gaël Richard
TL;DR
Bandwidth extension seeks to reconstruct missing high-frequency content from a band-limited signal, and the paper asks whether multi-rate flexibility can be achieved with discrete codec tokens. It trains a decoder-side transformer to complete band-limited codes while leaving the codec frozen. The method performs competitively across input rates and codecs, with a stated exception at the narrowest out-of-domain rate.
Problem
The paper asks whether multi-rate bandwidth extension can be achieved inside a codec using discrete tokens, rather than exposing a generative model to raw band-limited audio.
Source: §1 · p. 1Method
A decoder-side transformer predicts full-band codec codes from band-limited codes while the codec remains frozen, using a common representation across input rates.
Source: §1 · p. 1, §2.1 · p. 2, §2.3 · p. 2Results
On music mixtures, the method exceeds A2SB and UniverSR at every rate; out of domain, it leads except at 8 kHz, where listeners rate it close to A2SB.
Source: §5 · p. 4Takeaways & Limitations
The same token-completion recipe transfers between spectral-domain and waveform-domain codecs, so it does not require tokens to carry a frequency axis.
Source: §4.2 · p. 3, §5 · p. 4The test sets are not neutral: UniverSR trained on MUSDB18 without a specified split, and OrchideaSOL is closer to its solo-instrument domain than to the paper’s music-mixture domain.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Bandwidth extension, the task of reconstructing the high-frequency components of an audio signal from its low-passed counterpart, is a long-standing problem in audio processing. In this work, we extend recent advances in neural architectures by framing bandwidth extension as an audio token prediction problem. Specifically, we train a transformer-based language model on the discrete representations produced by a disentangled neural audio codec, where the disentanglement is guided by a Harmonic-Percussive decomposition of the input signals, highlighting spectral structures particularly relevant for bandwidth extension. Our approach introduces a novel codec design that explicitly accounts for the downstream token prediction task, enabling a more effective coupling between codec structure and transformer modeling. This joint design yields high-quality reconstructions of the original signal, as measured by both objective metrics and subjective evaluations. These results highlight the importance of aligning codec disentanglement and representation learning with the generative modeling stage, and demonstrate the potential of global, representation-aware design for advancing bandwidth extension.
1. INTRODUCTION
The paper asks whether multi-rate bandwidth extension can be achieved inside a codec using discrete tokens. It freezes the codec and trains one transformer across input rates, reporting competitive results across codecs and domains.
1. INTRODUCTION
- The paper situates its question amid neural BWE systems that support multiple input rates by exposing generative models to raw band-limited signals.Earlier approaches include convolutional regressors, diffusion and flow-matching models [6], and hybrid differentiable-DSP methods.Source: §1 · p. 1
- Earlier codec-branch methods improved predictability of high-branch tokens but each branch ran at its own sampling rate and could not directly accept other input rates.Prior work used frequency-dependent or harmonic-percussive sections [16, 17].Source: §1 · p. 1
- The codec remains frozen while a decoder-side transformer predicts only content above the input’s Nyquist frequency, preserving the true low-frequency content.A common 48 kHz codec representation makes the transformer the only rate-dependent component, so one model can support multiple input rates without codec retraining.Source: §1 · p. 1
- One transformer uses codec tokens to predict full-band codes across 8, 16, 24, and 32 kHz inputs without receiving the cutoff frequency.The bandwidth is recoverable from the codes.Source: §1 · p. 1
- On music mixtures, the method outperforms A2SB [6] and UniverSR at every input rate; out of domain, it leads at all rates except 8 kHz.At 8 kHz, listeners rate the DAC predictor comparably to A2SB, the stronger external baseline.Source: §1 · p. 1
2. BANDWIDTH EXTENSION AS TOKEN COMPLETION
The method freezes a codec and predicts full-band discrete codes from band-limited codes, synthesizing only the missing high band. A depth-autoregressive transformer processes codes from two codec types, and the output combines predicted high frequencies with the unchanged low band.
2.1. Task
- A frozen codec encodes each band-limited input, and a decoder-side predictor estimates its full-band codes to synthesize only frequencies above the cutoff, with no side information transmitted.The bitrate remains that of the band-limited codec codes.Source: §2.1 · p. 2
2.2. Application to two different codecs
- The method is instantiated on DAC, whose waveform-domain tokens have no frequency axis, and SpectroStream, whose spectral-domain tokens correspond to time-frequency regions.Both codecs use residual vector quantization and are retrained at 48 kHz on the paper’s data before being frozen.Source: §2.2 · p. 2
2.3. Predictor
- The Transformer predicts all time positions at each step, autoregressing over RVQ depth and conditioning on band-limited levels plus previously predicted full-band levels.A learned step embedding identifies the RVQ level being predicted.Source: §2.3 · p. 2
- One model trains across all four input rates without being given the cutoff; training draws the rate uniformly per batch and uses teacher forcing with cross-entropy loss.The model is trained for 25k steps with AdamW at 10^-4 on a cosine schedule.Source: §2.3 · p. 2
2.4. Synthesis protocol
- Synthesis splices the ground-truth low band to the decoded prediction above the cutoff using a 1 kHz crossfade; only the first predicted level is decoded because deeper levels did not improve ViSQOL.The predictor is trained on all quantizer levels, and the splice is applied identically to all compared systems.Source: §2.4 · p. 2
3. EXPERIMENTAL SETUP
The evaluation uses music mixtures for primary testing and solo-instrument material for out-of-domain testing, alongside objective metrics and a listening test. The authors note that training-set overlap makes neither test set neutral.
3.1. Data
- Predictors train on a weighted mixture of Jamendo, MedleyDB, MUSDB18 training data, and ENST-Drums; the primary test uses 1,000 MUSDB18 excerpts and the out-of-domain test uses 1,000 OrchideaSOL excerpts.Jamendo files are filtered to retain those with effective bandwidth reaching at least 19 kHz.Source: §3.1 · pp. 2–3
- Neither test set is neutral: UniverSR trained on MUSDB18 without a specified split, while A2SB never saw it; OrchideaSOL is closer to UniverSR’s solo-instrument domain than to the music-mixture training domain.OrchideaSOL is out of training for all three systems.Source: §3.1 · p. 3
3.2. Metrics
- ViSQOL-audio is the main objective metric, supplemented by multi-resolution mel and STFT distances, waveform ℓ1, and SI-SDR.SI-SDR is reported but not used to discriminate models because it is sensitive to sample-level misalignment.Source: §3.2 · p. 3
3.3. Baselines and listening test
- A2SB is a 44.1 kHz diffusion Schrödinger bridge over spectrograms, while UniverSR is a 48 kHz vocoder-free conditional flow-matching model supporting 8, 12, 16, and 24 kHz inputs.The comparison is not strictly like-for-like: the predictor uses quantized 12 kbit/s tokens, while the baselines use the raw low-band signal.Source: §3.3 · p. 3
- The MUSHRA test evaluates the hardest 8 kHz input setting, comparing the DAC predictor with A2SB on 12 MUSDB18 excerpts using ten non-expert listeners.The excerpt selection covers both high and lower concentrations of energy above 8 kHz.Source: §3.3 · p. 3
4. RESULTS
Both predictors lead on MUSDB18 across all tested rates and objective metrics, while the out-of-domain and listening results qualify that advantage. Results across two codecs and predictor-design tests show multi-rate completion without an explicit rate embedding, with bandwidth information concentrated at the first codebook level.
4.1. Comparative evaluation
- On MUSDB18, both predictors beat both baselines at every input rate on all three objective metrics; on OrchideaSOL, they lead at 16–32 kHz, while UniverSR leads all metrics at 8 kHz.The objective metrics order the four systems consistently on MUSDB18; the OrchideaSOL results also show generalization to unseen material at wider inputs.Source: §4.1 · p. 3, Table 1 · p. 4
- At 8 kHz, listeners rated the DAC predictor comparably to A2SB, despite A2SB’s three-point median advantage and overlapping interquartile ranges.The ViSQOL advantage did not carry over to listeners; the authors attribute this to fuller high-band filling accompanied by slight artifacts.Source: §4.1 · p. 3, Table 2 · p. 4
4.2. Discussing the results with the two different codecs
- The same recipe works with DAC’s waveform-domain tokens, which have no frequency axis, as well as with SpectroStream’s spectral-domain tokens.The authors conclude that the method exploits the completion task, not a codec-specific token property.Source: §4.2 · p. 3
- DAC captures more available headroom than SpectroStream at every rate (53.5/65.3/69.1/69.3% versus 47.4/58.0/62.6/57.6%), though excerpt lengths differ.DAC uses 2.5-second excerpts versus 5 seconds for SpectroStream; shorter excerpts cost quality, while DAC still has the longer token sequence.Source: §4.2 · p. 3
4.3. Predictor design
- One multi-rate predictor covers four input rates within 0.03 ViSQOL of four rate-specific predictors at most; adding a cutoff embedding brings no benefit.All trained predictors improve over the band-limited anchor, with gains of +1.40, +1.48, +0.96, and +0.31 ViSQOL as input rate widens.Source: §4.3 · p. 3, Table 3 · p. 4
- Under a narrowband-noise probe, DAC’s level-1 codebook carries more frequency information than SpectroStream’s, but SpectroStream’s entries are more frequency-ordered.Corrected mutual information is 3.04 bits for DAC and 1.82 bits for SpectroStream; entry-distance correlation is 0.218 and 0.438, respectively. The ordering gap remains when both use their 32 most frequent entries.Source: §4.3 · p. 4
- Bandwidth is legible at level 1 but essentially nowhere deeper, explaining why no rate embedding is needed and why one decoded level suffices.The less frequency-ordered DAC codebook captures more headroom at every rate, so frequency ordering itself is not why completion works.Source: §4.3 · p. 4
5. CONCLUSION
Bandwidth extension is framed as decoder-side token completion, leaving the codec untouched and requiring no supplied cutoff frequency. The approach transfers between spectral- and waveform-domain codecs and exceeds A2SB and UniverSR on music mixtures at every rate while reading a 12 kbit/s token stream.
5. CONCLUSION
- Bandwidth extension is decoder-side token completion that leaves the codec untouched and does not require supplying the cutoff frequency.Codebook probing finds input bandwidth legible in level-1 entries of both codecs and essentially nowhere deeper, which explains why decoding one level is sufficient.Source: §5 · p. 4
- The approach transfers unchanged between spectral- and waveform-domain codecs and exceeds A2SB and UniverSR on music mixtures at every rate while reading a 12 kbit/s token stream.At 8 kHz, listeners rate it close to A2SB; out of domain, it leads at every rate except 8 kHz. Training one predictor across several codecs is identified as a natural continuation.Source: §5 · p. 4
- The predictor transfers between codec domains without relying on tokens that carry a frequency axis.The reported transfer is from a spectral-domain codec to a waveform-domain codec.Source: §5 · p. 4