PhaseGAN: High-Fidelity Vocoder via Decoupled Amplitude and GAN-Driven Phase Reconstruction

1 Introduction

• GAN-based vocoders are described as balancing inference speed, synthesis quality, and versatility for practical applications, while training instability and artifacts remain challenges.The related-work discussion also notes real-time synthesis efficiency and competitive quality alongside those challenges.Source: §1 · p. 2
• Amplitude reconstruction is treated as interpolation because mel features contain amplitude information, whereas phase reconstruction is generative because the input lacks phase and one amplitude can match multiple perceptually similar phases.The authors propose separate methods for these tasks, with phase conditioned on estimated amplitude.Source: §1 · p. 2
• The amplitude stage interpolates Mel spectra to the linear-frequency scale, then uses ICCRN, which avoids frequency-axis down- and up-sampling to reduce artifacts, parameters, and computation.The paper says ICCRN can mitigate high-frequency artifacts.Source: §1 · p. 3
• PhaseGAN is reported to outperform existing approaches in model size, computational efficiency, and synthesized audio quality, with generalization to unseen speakers and Chinese singing.Experiments on end-to-end speech synthesis also confirm effectiveness in that task.Source: §1 · p. 3
• For phase reconstruction, PhaseGAN uses R3-GAN adversarial training to generate phase conditioned on amplitude without labeled phase information, addressing the one-to-many mapping.The paper reports greater training stability for R3-GAN in this setting.Source: §1 · p. 3

2.1 Overview

• Mel-filterbank pseudo-inversion, ICCRN amplitude recovery, conditioned phase generation, and inverse Fourier transformation form the four-step inference pipeline.Source: §2.1 · p. 3

2.2 ICCRN

• ICCRN avoids frequency downsampling and upsampling, keeps the frequency dimension at 513, and uses a unidirectional time-axis LSTM for causal modeling.It also applies cepstral analysis and LSTMs along frequency and time axes for dual-path modeling.Source: §2.2 · pp. 3–4

2.3 Amplitude Spectrum Reconstruction

• An ICCRN maps the pseudo-inverted Mel spectrum to an optimized amplitude spectrum, with Softplus in the output layer and MSE estimation loss.The pseudo-inverted input has frequency dimension 513; the loss compares target and estimated amplitude spectra.Source: Network · pp. 4–5

2.4 Phase Spectrum Generation

• The phase generator produces pseudo-real and pseudo-imaginary components in parallel, which Φ converts into a phase spectrum restricted to (-π,π].Source: §2.4 · p. 5
• The discriminator takes time-domain waveforms through parallel multi-scale modules, and its loss includes adversarial terms and zero-centered gradient penalties.The penalties use a dynamic decay coefficient and gradients with respect to real and generated samples.Source: Training Loss · p. 6
• Phase generation uses R3-GAN, while the generator objective combines multi-resolution STFT and adversarial losses to support phase-amplitude consistency.The multi-resolution STFT term indirectly constrains phase gradients during training.Source: Training Loss · pp. 5–6

3.1 Datasets

• LJSpeech provides the single-speaker benchmark, while VCTK and Opencpop test unseen-speaker and singing generalization without fine-tuning.The models are trained on LJSpeech; VCTK tests two speakers and Opencpop supplies Mandarin singing clips.Source: Single speaker speech synthesis · p. 7, Unseen speaker speech synthesis · p. 7, Singing Voice speech synthesis · p. 7

3.2 Baseline

• The benchmark includes HiFi-GAN, iSTFTNet, APNet, BigVGAN, APNet2, Vocos, and FreeV, retrained on the VITS codebase splits.Baseline parameters follow their officially released optimal versions.Source: §3.2 · pp. 7–8

3.3 Evaluation Metrics

• Evaluation combines crowd-sourced 5-scale MOS ratings with objective metrics for pitch accuracy, perceptual quality, spectral fidelity, and intelligibility.Twenty listeners rated samples; objective measures include pitch and F0 RMSE, UTMOS, WB-PESQ, MCD, STOI, periodicity error, and V/UV F1.Source: Subjective evaluation · p. 8, objective evaluation · p. 8

3.4 Training Setup

• Both proposed stages use 16-sample minibatches on LJSpeech, with Stage 1 trained for 1 million steps and Stage 2 for 1.5 million.The study trains standard and lightweight PhaseGAN variants, with differing ICCRN channel counts and five CFB layers across models.Source: §3.4 · pp. 8–9

4.2 Ablation Study

• Removing MRSTFT loss, zero-centered gradient penalty, adversarial loss, or either ICCRN core module degrades reported speech-synthesis performance or quality.The authors associate MRSTFT removal with poorer phase and amplitude fitting, zero-centered gradient penalty removal with diminished convergence, and adversarial-loss removal with mechanical artifacts and lower UTMOS.Source: §4.2 · pp. 9–10

4.1 Single-Speaker Expressiveness

• PhaseGAN (1.6M parameters) achieves the reported best UTMOS (4.238) and MCD (2.108), while PhaseGAN-s has near-SOTA quality with over 90% fewer parameters.The authors also report the best subjective MOS for PhaseGAN and lowest periodicity for PhaseGAN (0.088) and PhaseGAN-s (0.112).Source: §4.1 · p. 9, Table 1 · p. 10

4.3 Generalization to Unseen Speakers

• Without fine-tuning on VCTK, PhaseGAN outperforms HiFi-GAN and FreeV across multiple metrics and is reported to improve clarity and naturalness for unseen speakers.The authors infer that phase generation primarily depends on amplitude-spectrum information rather than speaker-specific characteristics.Source: §4.3 · p. 10, Table 2 · p. 11

4.4 Generalization to Singing Voice

• In zero-shot transfer from English LJSpeech speech to Mandarin Opencpop singing, PhaseGAN reconstructs clear, complete harmonics, while baseline samples have blurred harmonic structures.The reported comparison spans differences in language, articulation mode, and acoustic distribution.Source: §4.4 · pp. 10–11, Table 3 · p. 11

4.5 End-to-End Speech Synthesis

• With pre-trained vocoders and no fine-tuning in the LJSpeech pipeline, PhaseGAN is reported to show high compatibility with acoustic-model-driven synthesis and virtually no high-frequency artifacts.The authors also report clearer, more complete low-frequency harmonics than the baseline.Source: §4.5 · pp. 11–12, Table 4 · p. 12

• PhaseGAN introduces a phase-generation mechanism inspired by R3GAN to address vocoder phase reconstruction, which the authors describe as the first successful GAN-based solution to this problem.Source: §5 · p. 12
• The authors report superior synthesis quality over state-of-the-art publicly available models and exceptional generalization capabilities.Source: §5 · p. 12
• At complexity comparable to FreeV, PhaseGAN uses substantially fewer parameters while delivering superior audio fidelity, supporting its potential for low-latency, low-memory edge synthesis and industrial deployment.Source: §5 · p. 12

• The authors plan to validate the method on more diverse datasets with a wider variety of speakers and languages.Source: §6 · p. 12
• Future work will test the phase reconstruction method on different models to assess its universality.Source: §6 · p. 12
• The authors plan to extend phase reconstruction to codec, speech enhancement, and bandwidth extension applications.Source: §6 · p. 12