Source-linked AI summary
Nonparallel High-Quality Audio Super Resolution with Domain Adaptation and Resampling CycleGANs
Reo Yoneyama, Ryuichi Yamamoto, Kentaro Tachibana
TL;DR
Neural audio super-resolution models trained on paired signals can degrade on out-of-domain inputs, while target-domain high-resolution data are often unavailable. Dual-CycleGAN connects domain adaptation and resampling CycleGANs to use nonparallel data. The paper reports outperforming conventional methods in two challenging SR settings without paired data.
Problem
Paired-data neural audio SR degrades on out-of-domain inputs, and target-domain HR audio is generally unavailable for conventional methods requiring parallel LR–HR data.
Source: §1 · p. 1Method
Dual-CycleGAN connects CycleGAN-based domain adaptation and resampling processes to learn an LR–HR mapping using nonparallel and parallel data.
Source: §1 · p. 1, §2.1 · p. 2Results
Takeaways & Limitations
The paper identifies crosslingual SR as a valuable direction for low-resource languages lacking sufficient studio-quality data.
Source: §4 · p. 4Handling input speech from arbitrary domains through any-to-one domain adaptation remains a future direction.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Neural audio super-resolution models are typically trained on low- and high-resolution audio signal pairs. Although these methods achieve highly accurate super-resolution if the acoustic characteristics of the input data are similar to those of the training data, challenges remain: the models suffer from quality degradation for out-of-domain data, and paired data are required for training. To address these problems, we propose Dual-CycleGAN, a high-quality audio super-resolution method that can utilize unpaired data based on two connected cycle consistent generative adversarial networks (CycleGAN). Our method decomposes the super-resolution method into domain adaptation and resampling processes to handle acoustic mismatch in the unpaired low- and high-resolution signals. The two processes are then jointly optimized within the CycleGAN framework. Experimental results verify that the proposed method significantly outperforms conventional methods when paired data are not available. Code and audio samples are available from https://chomeyama.github.io/DualCycleGAN-Demo/.
1. INTRODUCTION
Paired-data neural audio super-resolution degrades on out-of-domain inputs, while target-domain high-resolution audio is generally unavailable for training. Dual-CycleGAN combines domain adaptation and resampling to use nonparallel data, and is reported to outperform conventional methods in two challenging settings.
1. INTRODUCTION
- Paired-data neural audio SR degrades when input acoustics differ from training data, limiting use on out-of-domain speech.Examples include speech recorded in different environments and TTS-generated speech.Source: §1 · p. 1
- Target-domain high-resolution audio is generally unavailable, making conventional methods that require parallel LR–HR pairs difficult to apply.The paper identifies this as a constraint on addressing out-of-domain issues.Source: §1 · p. 1
- Dual-CycleGAN uses two connected CycleGANs to decompose SR into domain adaptation and resampling, jointly optimized through joint pre-training and fine-tuning.It learns an implicit LR–HR mapping using nonparallel and parallel data, including when target-domain HR signals are unavailable.Source: §1 · p. 1
- Dual-CycleGAN is reported to outperform conventional methods for SR across recording environments and on TTS-generated speech.The paper distinguishes its high-quality 16-to-48 kHz setting from concurrent speaker-verification work on 8-to-16 kHz telephone signals.Source: §1 · p. 1
2. PROPOSED METHOD
Dual-CycleGAN composes domain adaptation from a source LR domain to a target LR domain with resampling from target LR to target HR. Joint pre-training and fine-tuning optimize the connected CycleGANs, with perceptual cycle consistency and specialized network components addressing the SR setting.
2.1. Overview
- Inference composes domain adaptation from SLR to TLR with resampling from TLR to THR, using G3(G1(·)); the two CycleGANs share D2 to save parameters.The resampling CycleGAN learns LR–HR conversion within target domain T, where parallel data are available.Source: §2.1 · pp. 1–2
2.2.1. Joint pre-training
- Joint pre-training simultaneously trains both conversions with least-squares GANs, and its adversarial, cycle-consistency, and identity losses are used for stable training.The generator losses are weighted 1, 10, and 10, respectively, for adversarial, cycle-consistency, and identity terms.Source: §2.2.1 · p. 2
2.2.2. Fine-tuning
- Fine-tuning replaces three pre-training losses and introduces a cycle-consistency loss over the full four-generator cycle to optimize the composite mappings.The paper says this direct optimization eases the effect of domain shift between real and fake TLR samples induced by imperfect domain adaptation.Source: §2.2.2 · p. 2
2.3. Perceptual cycle consistency
- Mel-spectrogram L1 cycle and identity losses accommodate SR’s ill-posedness, allowing flexibility in high-frequency bands for cycle-consistent conversions.The paper contrasts this perceptual loss with waveform-domain L1, which it says makes learning bijective cycle-consistent functions difficult when multiple HR waveforms correspond to one LR waveform.Source: §2.3 · p. 2
2.4. Network architecture
- Generators use a WaveCycleGAN2-based architecture with 1D convolutions and GLUs; G3 and G4 add sinc-based interpolation for resampling.Discriminators operate on waveform and log-amplitude linear spectral domains, with multi-band grouped spectral discriminators.Source: §2.4 · p. 3
3. EXPERIMENTAL EVALUATIONS
The evaluation covers audio SR across recording environments and on TTS-generated speech, with dataset and model configurations described before results from those experiments.
3. EXPERIMENTAL EVALUATIONS
- The evaluation comprises SR across different recording environments and SR on TTS-generated speech.The section introduces dataset and model configurations before reporting the two experiment results.Source: §3 · p. 3
3.1. Experimental setups
Experiments use LJ Speech as nonparallel source-domain LR data and VCTK as parallel target-domain data. The setup also trains paired-data baselines and a parameter-matched single-CycleGAN comparison.
3.1.1. Database
- LJ Speech supplies approximately 24 hours of single-speaker 16 kHz source-domain LR data, while VCTK supplies approximately 44 hours of 48 kHz speech and paired 16/48 kHz target-domain data.VCTK includes 110 English speakers; the datasets were split into training, validation, and test sets.Source: §3.1.1 · p. 3
3.1.2. Model details
- Dual-CycleGAN training used 400 K joint-pretraining iterations followed by 200 K fine-tuning iterations, with minibatches of four and 12 K-sample clips.The setup used Adam, weight normalization, and gradient-norm clipping at ten.Source: §3.1.2 · p. 3
- HiFi-GAN+ and WSRGlow served as baselines trained only on VCTK because they cannot use unpaired data; a parameter-matched single CycleGAN tested explicit domain adaptation.The single CycleGAN first used VCTK parallel data for 100 K iterations to calculate identity loss, then trained for 500 K iterations using all data.Source: §3.1.2 · p. 3
3.2. SR across different recording environments
In cross-recording-environment SR, Dual-CycleGAN significantly outperformed the baseline SR methods, and ablations supported the fine-tuning procedure and mel-spectral losses.
3.2.1. Comparison with baselines
- Twenty subjects evaluated randomly paired samples generated from LJ Speech 16 kHz inputs, with 200 samples evaluated per method pair.The test also included recorded 22.05 kHz samples for comparison.Source: §3.2.1 · p. 3
- Dual-CycleGAN significantly outperformed HiFi-GAN+ and WSRGlow, although it was inferior to the 22.05 kHz recorded speech.The authors hypothesize that noise in some VCTK data contributed to the gap with recorded speech.Source: §3.2.1 · pp. 3–4, Table 1 · p. 4
3.2.2. Ablation study
- Fine-tuning improved speech quality over joint pre-training, and the comparison supported mel-spectral cycle-consistency and identity-mapping losses.The ablation used a jointly pre-trained model and a model trained with waveform-domain cycle-consistency and identity losses as comparisons.Source: §3.2.2 · p. 4, Table 2 · p. 4
3.3. SR on TTS-generated speech
The experiments applied Dual-CycleGAN to TTS-generated speech using recorded and synthetic data, then compared sound-quality ratings across enhanced TTS systems and reference systems.
3.3.1. TTS setup
- Dual-CycleGAN was trained with recorded and synthetic speech, while HiFi-GAN+ and WSRGlow used recorded data only.The TTS system used VITS models trained on 16 kHz and 48 kHz VCTK speech.Source: §3.3.1 · p. 4
3.3.2. Subjective listening tests
- Twenty subjects rated 20 randomly selected test samples across enhanced TTS systems, VITS reference systems, and 48 kHz clean recordings.Subjects judged the sound quality of entire utterances.Source: §3.3.2 · p. 4
- 4.51 was the proposed TTS system's best score among enhanced TTS systems, significantly outperforming the two baseline SR methods.The tests also found that a system trained on 16 kHz speech could be improved significantly to match the quality of a system trained on 48 kHz speech.Source: §3.3.2 · p. 4, Table 3 · p. 4
4. CONCLUSION
Dual-CycleGAN performed high-quality nonparallel audio SR across domains and outperformed baseline SR methods in two situations where parallel data were unavailable.
4. CONCLUSION
- Dual-CycleGAN outperformed baseline SR methods in two challenging situations where parallel data were unavailable.The authors identify arbitrary-domain adaptation and crosslingual SR for low-resource languages without sufficient studio-quality data as future directions.Source: §4 · p. 4