PaperlayerView exact revision on arXiv

Source-linked AI summary

SCNet: Enhancing GAN-based Speech Generation with Subband Condition Network and Magnitude-aware Phase Loss

Nan Xu, Mingxue Yang

arXiv:2609.10025v1eess.AS

TL;DR

GAN-based vocoders can lose fine-grained spectral information when they lack initial-condition guidance. SCNet adds a subband condition network and magnitude-aware anti-wrapping phase loss, and the authors report superior subjective and objective speech quality. They also identify slower synthesis than Vocos as a limitation.

  • Problem

    GAN-based vocoders can lack initial-condition guidance, resulting in loss of fine-grained spectral information.

    Source: §1 · p. 1

  • Method

    SCNet integrates low-frequency subband Fourier priors predicted by CondNet into an iSTFT-based backbone and uses a magnitude-aware anti-wrapping phase loss.

    Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 3

  • Results

    SCNet reports superior subjective and objective speech quality and validates the effectiveness of CondNet and the anti-wrapping phase loss.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments support CondNet and the magnitude-aware anti-wrapping phase loss as effective components of the proposed vocoder.

    Source: §1 · p. 2, §6 · p. 8

    SCNet has slower synthesis speed than Vocos, and its use as a universal vocoder for speech, audio, and music has not been verified.

    Source: §5 · p. 8

Abstract

from arXiv · show

Recent speech generation has been predominantly driven by GAN-based networks aimed at high-quality waveform synthesis from mel-spectrograms. However, these methods often operate as black-box models, leading to the loss of inherent spectral information. In this work, we propose SCNet, a GAN-based vocoder augmented with a Subband Condition Network to address this issue. Specifically, SCNet leverages a subband signal predicted by a lightweight condition network as prior knowledge. This subband signal is then transformed via STFT to obtain Fourier coefficients, which are integrated into the backbone for the enhanced reconstruction. Additionally, to mitigate the phase wrapping, we introduce a magnitude-aware phase loss that computes instantaneous phase errors weighted by the corresponding magnitude, emphasizing regions with higher energy. Experimental results demonstrate that SCNet achieves superior performance in both objective and subjective evaluations for high-quality speech generation.

1. Introduction

Mel-spectrogram vocoders support speech synthesis, but GAN-based systems can lose fine-grained spectral information and underweight important phase errors. SCNet adds subband frequency priors and a magnitude-aware anti-wrapping phase loss, with experiments reporting strong speech quality and competitive inference speed.

1. Introduction

  • Mel-spectrograms are compact intermediate representations used in two-stage speech synthesis, including TTS, singing voice synthesis, and voice conversion.Traditional signal-processing mappings can introduce nonnegligible artifacts.Source: §1 · p. 1
  • GAN-based vocoders can lack initial-condition guidance and lose fine-grained spectral information, while pitch-tracking errors can limit neural source-filter approaches.The passage cites incorrect voiced/unvoiced flags and pitch halving or doubling as examples of pitch-estimation errors.Source: §1 · p. 1
  • Existing phase losses treat time-frequency bins equally, although phase wrapping and the perceptual importance of magnitude make that weighting inadequate.The paper notes that small phase errors at large-magnitude bins may receive less attention.Source: §1 · p. 1, Figure 1 · p. 2
  • SCNet combines a dual-branch GAN vocoder with low-frequency subband Fourier priors from CondNet and a magnitude-weighted anti-wrapping phase loss.The subband signal is transformed with STFT, and its frequency information is integrated into the backbone; raw magnitude values weight phase errors.Source: §1 · pp. 1–2
  • SCNet reports superior subjective and objective speech quality, validates CondNet and the anti-wrapping phase loss, and achieves competitive speed, especially against conventional GAN vocoders.The authors also report that synthesis is slower than Vocos in the limitations section.Source: §1 · p. 2

2. Related Work

Related vocoders generate waveforms directly or predict Fourier coefficients for iSTFT reconstruction, while phase-loss designs seek to model phase explicitly. The cited approaches differ in their use of pitch information, spectral structure, and phase weighting.

Direct waveform generation

  • Direct-generation GANs pursue high-fidelity, efficient waveform synthesis, but their time-frequency mel-spectrogram inductive bias is not well utilized.HiFiGAN uses multi-receptive field fusion, while Avocodo combines subband and collaborative multi-band discriminators.Source: §2 · p. 2

iSTFT-based generation

  • iSTFT-based vocoders predict full-band Fourier coefficients for waveform reconstruction, but the cited systems lack explicit supervision of those coefficients.HiNet separates amplitude and NSF-based phase prediction, whereas Vocos predicts magnitude and phase together.Source: §2 · p. 2

Phase loss

  • Phase-loss methods explicitly model phase, but the cited supervised losses ignore magnitude-based weighting across time-frequency bins.PhaseAug rotates frequency-bin phases; FA-GAN uses multi-resolution real and imaginary losses; APNet and APNet2 use explicit phase losses.Source: §2 · p. 2

3. Method

SCNet combines a waveform-generating backbone with a conditional branch that supplies subband frequency information, and uses a magnitude-aware phase loss to address phase wrapping. The method also specifies joint training objectives for both networks.

3.1. Overview

  • The backbone predicts magnitude and phase for full-band iSTFT reconstruction, while a conditional network supplies subband waveform information that is transformed by STFT and integrated through two coupling blocks.The networks are jointly optimized.Source: §3.1 · pp. 2–3

3.2. CondNet architecture

  • CondNet uses ConvNeXtV2 blocks to predict magnitude and phase in a low-frequency subband from mel-spectrograms, then converts the predictions into a subband waveform with inverse STFT.The inverse STFT uses hop size 64; the resulting waveform has a 6 kHz sampling rate experimentally.Source: §3.2 · p. 3
  • Aligned subband frequency representations are added to corresponding backbone upsample-layer outputs as prior condition information.They are produced with STFT at hop sizes 4 and 1 and supplied to coupling blocks 1 and 2, respectively.Source: §3.2 · p. 3

3.3. Magnitude-aware phase loss

  • The phase loss uses the true wrapped phase error and weights it by target magnitude, giving more attention to small phase errors at high-magnitude bins.The true phase error is restricted to (−π, π], addressing errors caused by phase wrapping.Source: §3.3 · p. 4

3.4. Training objectives

  • Training combines adversarial, feature-matching, and reconstruction losses; adversarial and feature-matching losses apply only to the backbone, while reconstruction losses apply to both networks.Reconstruction includes mel-spectrogram and phase losses, with λm = 45 and λpha = 45.Source: §3.4 · p. 4

4.1. Datasets

The study trains on LibriTTS train-clean-100 and evaluates on separate in-domain and out-of-domain test sets.

4.1. Datasets

  • Training uses 24 kHz LibriTTS train-clean-100 speech with 80-bin log mel-spectrograms spanning 0–12 kHz.Source: §4.1 · p. 4
  • Evaluation uses 500 LibriTTS utterances for the in-domain set and 500 VCTK utterances to test generalization to unseen speakers.Source: §4.1 · p. 4

4.2. Training setups

SCNet combines a low-rate subband waveform branch with a CondNet that predicts subband Fourier coefficients. Models are trained for up to 1M steps, with ablations trained for 0.5M steps.

4.2. Training setups

  • The study reports separate result tables for in-domain/out-of-domain speech, LibriTTS development subsets, and MOS comparisons.Source: Table 2 · p. 5, Table 3 · p. 5, Table 4 · p. 5
  • CondNet predicts subband Fourier coefficients, then inverse STFT generates a low-frequency subband waveform; the backbone uses inverse STFT to generate the final waveform.The subband waveform is sampled at 6 kHz; the backbone uses two 8× upsample blocks.Source: §4.2 · pp. 4–5
  • Training uses 24,576-sample segments, batch size 16, AdamW, and an initial generator and discriminator learning rate of 2e-4, decayed exponentially by 0.999.All models train for up to 1M steps; ablation models train for 0.5M steps.Source: §4.2 · p. 5

4.3. Baselines and evaluations

SCNet is evaluated against direct-waveform and iSTFT-based vocoders using objective speech metrics, subjective MOS ratings, and synthesis-speed measurements.

4.3. Baselines and evaluations

  • Baselines include HiFi-GAN and BigVGAN for direct waveform generation, plus iSTFTNet, HiFTNet, and Vocos for iSTFT-based generation.Baselines are trained with SCNet configurations, with iSTFTNet trained using an unofficial implementation.Source: §4.3 · p. 5
  • Objective evaluation uses PESQ, multi-resolution STFT distance, periodicity error, voiced/unvoiced F1, and F0 root mean square error.Source: §4.3 · pp. 5–6
  • Subjective evaluation uses 20 raters scoring speech naturalness from 1 to 5; synthesis speed is measured on a V100 and reported as xRT.An xRT value of 1.0 denotes real-time speed; Table 5 reports training and synthesis speeds and model footprints.Source: §4.3 · p. 6, Table 5 · p. 6

4.4. Results

SCNet outperforms GAN-based baselines on the reported speech evaluations, including unseen-speaker tests, while using substantially less model size than BigVGAN. Ablations associate its gains with CondNet prior information and magnitude-aware phase loss.

4.4.1. Model performance

  • SCNet outperforms same-parameter GAN baselines on reported metrics and beats 112M-parameter BigVGAN in-domain and out-of-domain with approximately 1/8 its model size.The authors also report consistently improved metric scores on the out-of-domain set with unseen speakers.Source: §4.4.1 · p. 6
  • With 2M training steps on LibriTTS train-full-960, SCNet achieves competitive performance with BigVGAN on all metrics; at 1M steps it is better on most metrics but has a slight pitch-metric decline.Source: §4.4.1 · p. 6

4.4.2. TTS experiments

  • SCNet achieves a 4.09 MOS and outperforms other baseline vocoders in the reported CosyVoice-based TTS evaluation.Source: §4.4.2 · p. 6

4.4.3. Analysis of feature matching loss

  • Feature-matching loss rises before stabilizing without conditional prior information, whereas it decreases before stabilizing with SCNet’s prior-conditioned backbone.Figure 5 compares SCNet, SCNet without CondNet, and HiFiGAN under the same MPD and MRD discriminators, trained up to 1M steps.Source: §4.4.3 · p. 7, Figure 5 · p. 7

4.4.4. Training and synthesis speeds

  • SCNet synthesizes slightly faster than HiFiGAN and about 4× faster than BigVGAN, but is more than 4× slower than Vocos.It uses less training memory than baselines other than Vocos, and BigVGAN takes over 3× as many training days as SCNet for 1M steps.Source: §4.4.4 · p. 7

4.4.5. Ablation study

  • Predicting a full-band rather than subband waveform with CondNet degrades performance, and omitting the condition network causes a marked decline across metrics.Omitting the two STFT operators also slightly degrades quality; the authors attribute this to misalignment between time-domain prior information and frequency-domain magnitude and phase.Source: §4.4.5 · p. 7
  • Removing magnitude-aware phase weighting leads to non-convergent phase loss, while the proposed weighted loss declines quickly during training.Dropping the phase loss or replacing it with APNet2’s phase loss also reduces performance; omitting the magnitude weight performs similarly to omitting the loss.Source: §4.4.5 · p. 8, Figure 6 · p. 8
  • Different phase-loss weights yield approximate results, while a fixed 1M-step CondNet significantly outperforms a fixed 0.3M-step CondNet.The authors report that better subband-waveform quality provides better prior guidance to the backbone.Source: §4.4.5 · p. 8, Table 9 · p. 8

5. Limitations

SCNet synthesizes more slowly than Vocos, and its use as a universal vocoder for speech, audio, and music remains unverified.

5. Limitations

  • SCNet has slower synthesis speed than Vocos.The authors suggest that a more flexible condition-network and backbone framework could improve speed.Source: §5 · p. 8
  • SCNet has not been verified as a universal vocoder for speech, audio, and music generation.The authors identify this scope as future work.Source: §5 · p. 8
  • The authors identify a more flexible framework combining the condition network and backbone branch as a possible route to faster synthesis.Source: §5 · p. 8

6. Conclusion

SCNet combines a subband-prediction network with an iSTFT-based backbone and a magnitude-aware anti-wrapping phase loss. The authors report that the model and loss are effective and provide a new idea for neural vocoders.

6. Conclusion

  • CondNet predicts a subband waveform whose STFT yields aligned frequency-domain representations that provide prior information and simplify model training.Source: §6 · p. 8
  • The magnitude-aware anti-wrapping phase loss weights training by magnitude to help the network learn speech phase structures more correctly.Source: §6 · p. 8
  • The authors report that experiments demonstrate the effectiveness of SCNet and its phase loss, presenting the model as a new idea for neural vocoders.Source: §6 · p. 8

7. Use of Generative AI Disclosure

The authors declare that no AI tools or language models were used in the paper's writing, data analysis, or other work.

7. Use of Generative AI Disclosure

  • The authors declare that no AI tools or language models were used in writing, data analysis, or any other part of the paper.They state that all content is their own intellectual effort and original work.Source: §7 · p. 9
Loading 2609.10025v1…