PaperlayerView exact revision on arXiv

Source-linked AI summary

NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates

Seungu Han, Junhyeok Lee

arXiv:2206.08545v2eess.AScs.LG

TL;DR

Audio super-resolution models typically require separate training for each input and target sampling-rate pair, although real-world input bandwidth varies. NU-Wave 2 combines diffusion with STFC and BSFT to generate high-resolution audio from varied sampling rates; it reports strong frequency-band results and substantially fewer parameters than WSRGlow.

  • Problem

    Prior audio super-resolution models fixed input and target sampling rates, requiring separate training for each pair despite variable input bandwidth in real-world applications.

    Source: §1 · p. 1

  • Method

    NU-Wave 2 is a diffusion upsampling model that uses STFC to generate harmonics and BSFT to condition input bandwidth in the spectral domain.

    Source: §1 · p. 1, §3 · p. 2, §3.2 · p. 3

  • Results

    Using 0.63% of WSRGlow’s parameters, NU-Wave 2 achieves close LSD and LSD-HF results and the best SNR and LSD-LF.

    Source: §6 · p. 4

  • Takeaways & Limitations

    The model targets a fixed high-resolution sampling rate from inputs of varied rates, a setting for which prior models required separate rate-pair training.

    Source: §1 · p. 1

    Evaluation covers specific commonly used sampling rates, although the authors state the model can upsample other rates.

    Source: §6 · p. 4

Abstract

from arXiv · show

Conventionally, audio super-resolution models fixed the initial and the target sampling rates, which necessitate the model to be trained for each pair of sampling rates. We introduce NU-Wave 2, a diffusion model for neural audio upsampling that enables the generation of 48 kHz audio signals from inputs of various sampling rates with a single model. Based on the architecture of NU-Wave, NU-Wave 2 uses short-time Fourier convolution (STFC) to generate harmonics to resolve the main failure modes of NU-Wave, and incorporates bandwidth spectral feature transform (BSFT) to condition the bandwidths of inputs in the frequency domain. We experimentally demonstrate that NU-Wave 2 produces high-resolution audio regardless of the sampling rate of input while requiring fewer parameters than other models. The official code and the audio samples are available at https://mindslab-ai.github.io/nuwave2.

1. Introduction

Earlier audio super-resolution models fixed the input and target sampling rates, while NU-Wave could generate 48 kHz audio but had difficulty generating vowel harmonics and some frequency bands. NU-Wave 2 addresses these gaps with a diffusion model using STFC and BSFT to generate high-quality audio from varied input sampling rates with fewer parameters than other models.

1. Introduction

  • Prior models fixed input and target sampling rates, requiring separate training for each pair when real-world input bandwidth varies.The paper calls this setting the general neural audio upsampling task: producing a fixed-rate high-resolution signal from inputs of various sampling rates.Source: §1 · p. 1
  • NU-Wave 2 is a diffusion model for general neural audio upsampling that uses STFC to address NU-Wave’s problems and BSFT to adapt to various sampling rates.The paper identifies NU-Wave’s limitations as difficulty generating vowel harmonics and low performance across frequency bands.Source: §1 · p. 1
  • NU-Wave 2 generates high-quality audio from varied input sampling rates with fewer parameters than other models.Its BSFT bandwidth-conditioning method operates in the spectral domain.Source: §1 · p. 1

2. Diffusion Modeling

This section describes the diffusion formulation used to generate audio, including its forward noise process, training objective, and reverse-time sampling. The model is trained to predict noise and uses DDIM updates at inference.

2. Diffusion Modeling

  • The forward process samples latent states over time, and the paper uses a variance-preserving diffusion process with a log-SNR noise schedule.The schedule is defined as log signal-to-noise ratio and decreases strictly with time; the stated parameterization sets α_t = cos(at + b) and σ_t = sin(at + b).Source: §2 · p. 1
  • Training minimizes an L1 noise-prediction loss, with the model receiving the noisy latent, low-resolution signal, bandwidth embedding, and log-SNR.Training samples times across the batch to cover [0, 1] more evenly; the low-resolution signal is formed by filtering, downsampling, and upsampling the source.Source: §2 · pp. 1–2
  • The noise predictor is interpreted as an estimate of the latent log-density gradient, linking the denoising model to a probability-flow ODE.The paper describes DDIM as an integration rule for that ODE.Source: §2 · p. 2
  • At inference, the model uses DDIM updates to run the reverse process from noise toward the generated signal.The paper sets the number of sampling iterations to 8 for fast sampling and reports that larger iteration counts give similar results.Source: §2 · p. 2

3. Model Architecture

NU-Wave 2 extends NU-Wave with inputs that encode the noisy signal, low-resolution audio, bandwidth, and diffusion time. STFC supports harmonic generation, while BSFT conditions spectral features on input bandwidth.

3. Model Architecture

  • NU-Wave 2 takes noisy speech, low-resolution audio, log-SNR, and a one-hot bandwidth embedding as inputs; its 15 residual layers use 64 channels.The bandwidth embedding labels whether each band originates from the low-resolution signal.Source: §3 · p. 2, Figure 1 · p. 2

3.1. Short-time Fourier convolution (STFC)

  • STFC splits features between local convolution and a global spectral branch, then fuses their outputs.It uses STFT and iSTFT rather than FFT to handle audio with varying lengths; its receptive field supports periodic-signal and harmonic reconstruction.Source: §3.1 · pp. 2–3, Figure 2 · p. 3

3.2. Bandwidth spectral feature transform (BSFT)

  • BSFT uses bandwidth-conditioned scaling and shifting in the spectral domain to indicate which frequencies to conserve and which to generate.Its parameters come from a bandwidth one-hot embedding, and the authors report that the layer preserves low frequencies almost identically to ground truth.Source: §3.2 · p. 3

4. Experiments

The model is trained on VCTK for 48 kHz output with input rates sampled across a range, then evaluated at four fixed rates. Evaluation uses spectral and signal metrics, ABX judgments, and comparisons with two baselines.

4.1. Training

  • Training uses 44 hours of VCTK speech and samples input rates uniformly from 6–48 kHz for 48 kHz targets.Randomized Chebyshev Type I low-pass filtering is used during training; test inputs are 8, 12, 16, and 24 kHz.Source: §4.1 · p. 3

4.2. Evaluation

  • Evaluation measures SNR, LSD, LSD-HF, and LSD-LF, and uses an ABX test to assess whether outputs are distinguishable from ground truth.The ABX test on Amazon Mechanical Turk examined almost 1,800 cases.Source: §4.2 · p. 3

4.3. Baselines

  • The baselines are NU-Wave and WSRGlow; NU-Wave 2 has 1.7M parameters, versus 3.0M for NU-Wave and 270M for adjusted WSRGlow.The authors note that baseline hyperparameters may not be fully optimized for this experimental setting.Source: §4.3 · p. 3

5. Results

NU-Wave 2 improves on NU-Wave across most quantitative cases and reconstructs high-frequency structure that NU-Wave misses. Its ABX accuracy is close to chance, which the authors interpret as outputs nearly indistinguishable from reference signals.

5. Results

  • In the 16 kHz spectrogram, NU-Wave 2 reconstructs valid high frequencies that NU-Wave cannot; for 8 kHz input, it generates harmonics where baselines leave aliases.The figures mark the input Nyquist frequency with a red line.Source: §5 · p. 3, Figure 3 · p. 4, Figure 4 · p. 4
  • 24.5% versus NU-Wave’s 40.0%: NU-Wave 2 achieves this average of input’s LSD and LSD-HF, while WSRGlow achieves 22.4%.The authors report that NU-Wave 2 outperforms NU-Wave in most cases, with comparable LSD-LF and SNR.Source: §5 · p. 3
  • For 8, 12, 16, and 24 kHz inputs, NU-Wave 2 improves SNR over WSRGlow by 0.2, 0.4, 1.4, and 1.8 dB, respectively.For LSD-LF, WSRGlow exceeds 63.7% of input on average, compared with 40.5% for NU-Wave 2.Source: §5 · pp. 3–4
  • ABX accuracy for NU-Wave 2 is 55.3%, 53.2%, 52.1%, and 52.1% across the four input rates, which the authors say is close to 50%.The authors conclude that these outputs are nearly indistinguishable from the reference signals.Source: §5 · p. 4

6. Discussion

NU-Wave 2 generates harmonics across frequency bands and performs favorably against WSRGlow with far fewer parameters. Its evaluation uses selected sampling rates, although the model is described as supporting other input rates.

6. Discussion

  • STFCs let NU-Wave 2 generate harmonics across all frequency bands and provide a high receptive field in early layers.The paper attributes perceptual quality and parameter efficiency to this receptive-field design.Source: §6 · p. 4
  • Using 0.63% of WSRGlow’s parameters, NU-Wave 2 achieves close LSD and LSD-HF results and the best SNR and LSD-LF.The table covers 8, 12, 16, and 24 kHz inputs.Source: Table 1 · p. 4, §6 · p. 4
  • NU-Wave 2’s ABX accuracies are close to 50%, indicating outputs nearly indistinguishable from reference signals.The paper notes that downsampled inputs already retain sufficient information in the dominant-energy audio range, limiting significant differences between models.Source: Table 2 · p. 4, §6 · p. 4
  • The model was evaluated at specific commonly used sampling rates, while the authors state it can upsample other rates as well.Further work could target restoring additional distortions with one diffusion model.Source: §6 · p. 4

Related papers

Loading 2206.08545v2…