Source-linked AI summary
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis
Hubert Siuzdak
TL;DR
Time-domain neural vocoders achieve strong results but rely on computationally intensive upsampling, while direct Fourier-spectrogram generation is hindered by phase recovery. Vocos generates complex STFT coefficients with implicit phase wrapping and inverse-Fourier waveform reconstruction, matching state-of-the-art quality while delivering substantially higher computational efficiency.
Problem
Fourier-based vocoding is appealing, but direct reconstruction of complex spectrograms remains difficult because phase modeling and recovery are challenging.
Method
Vocos is a GAN-based vocoder that generates complex STFT coefficients, uses unit-circle phase estimation, and reconstructs waveforms with inverse Fourier transforms.
Results
Vocos matches state-of-the-art audio quality while providing substantially greater computational efficiency than time-domain methods.
Takeaways & Limitations
Vocos demonstrates that Fourier-based neural vocoding can combine high audio quality with computational efficiency.
Abstract
from arXiv · showhide
Recent advancements in neural vocoding are predominantly driven by Generative Adversarial Networks (GANs) operating in the time-domain. While effective, this approach neglects the inductive bias offered by time-frequency representations, resulting in reduntant and computionally-intensive upsampling operations. Fourier-based time-frequency representation is an appealing alternative, aligning more accurately with human auditory perception, and benefitting from well-established fast algorithms for its computation. Nevertheless, direct reconstruction of complex-valued spectrograms has been historically problematic, primarily due to phase recovery issues. This study seeks to close this gap by presenting Vocos, a new model that directly generates Fourier spectral coefficients. Vocos not only matches the state-of-the-art in audio quality, as demonstrated in our evaluations, but it also substantially improves computational efficiency, achieving an order of magnitude increase in speed compared to prevailing time-domain neural vocoding approaches. The source code and model weights have been open-sourced at https://github.com/gemelo-ai/vocos.
1 INTRODUCTION
Neural vocoding has largely modeled waveforms in the time domain, while Fourier-based synthesis offers a structured alternative whose phase representation remains difficult to model. Vocos addresses this gap by generating complex STFT coefficients with phase wrapping and maintaining temporal resolution, achieving state-of-the-art quality and substantially higher speed.
- Time-domain neural vocoding became dominant through autoregressive and nonautoregressive waveform models, including WaveNet and GAN-based approaches.
- Fourier-based representations retain time-frequency structure, but direct complex-spectrogram reconstruction remains difficult because phase wraps periodically and is challenging to estimate.
- Vocos is a GAN-based vocoder trained to generate complex STFT coefficients instead of directly generating waveform samples.
- Vocos maintains the same feature temporal resolution across layers and reconstructs the waveform using the Inverse Fast Fourier Transform rather than transposed-convolution upsampling.
- Vocos estimates phase with a unit-circle activation that implicitly incorporates phase wrapping, while ConvNeXt blocks replace the need for typical dilated-convolution designs.
- Vocos matches state-of-the-art audio quality and provides over an order of magnitude greater speed than time-domain counterparts.
2 RELATED WORK
Prior work has advanced GAN-based waveform vocoders and phase-reconstruction methods, but Fourier-coefficient generation has remained less successful and difficult to train. Existing approaches also face aliasing, artifact, or representation limitations.
- GAN-based vocoders: GAN-based vocoders use adversarial and feature-matching objectives to generate waveforms quickly and in parallel.
- GAN-based vocoders: Standard time-domain GAN vocoders expand features with transposed convolutions, a design susceptible to aliasing artifacts.
- Phase and magnitude estimation: Traditional and neural phase-reconstruction methods address spectrogram phase, but Griffin-Lim can introduce unnatural speech artifacts and direct phase modeling remains challenging.
- Phase and magnitude estimation: Many phase-and-magnitude methods assume full-scale magnitude spectrograms, whereas modern synthesis pipelines often use compact mel-spectrograms or pretrained latent features.
- Phase and magnitude estimation: Prior models that generate both STFT magnitude and phase have included implicit real-imaginary prediction and explicit parameterization, but some encountered training instability.
3 VOCOS
Vocos uses a Fourier-based generator that preserves temporal resolution and reconstructs waveforms with inverse STFT, while parameterizing magnitude and phase to produce complex coefficients. Its ConvNeXt backbone and GAN losses support this architecture.
- Overview: Vocos targets a Fourier-based time-frequency distribution and eliminates transposed convolutions from the generator.
- Overview: The architecture maintains temporal resolution throughout the network and performs waveform reconstruction solely with fast inverse STFT.
- STFT representation: STFT represents audio by applying Fourier transforms to successive overlapping, windowed signal frames.
- Backbone: The generator backbone adapts ConvNeXt with depthwise convolutions, inverted bottlenecks, GELU activations, and layer normalization.
- Head: Because real-valued signals have conjugate-symmetric Fourier transforms, Vocos predicts a single-side spectrum with nfft/2 + 1 coefficients per frame.
- Head: Vocos converts hidden outputs into magnitude and phase, computes M = exp(m), maps phase values onto the unit circle, and forms STFT = M · (x + jy).
- Head: The unit-circle parameterization ensures phase angles are correctly wrapped into (−π, π].
- Loss: Training uses multi-period and multi-resolution discriminators with reconstruction, adversarial, and feature-matching losses.
4 RESULTS
Vocos is evaluated against established vocoders on objective, subjective, out-of-distribution, codec-reconstruction, and efficiency measures. It generally matches or exceeds competing systems while offering substantial speed advantages.
- Objective evaluation: Vocos achieves the highest VISQOL and PESQ scores among the evaluated models and mitigates periodicity issues associated with time-domain GANs.BigVGAN slightly outperforms Vocos on UTMOS.
- Subjective evaluation: Vocos performs on par with the state of the art in perceived quality and similarity, with no significant MOS or SMOS difference from BigVGAN.The Wilcoxon signed-rank tests report p-values greater than 0.05.
- Out-of-distribution evaluation: Vocos consistently achieves the highest VISQOL scores across vocals, drums, bass, other instruments, and mixtures in MUSDB18 evaluation.The evaluation tests generalization to unseen acoustic conditions.
- Out-of-distribution evaluation: Vocos provides more accurate upper-frequency harmonic reconstruction than BigVGAN and HiFi-GAN on an out-of-distribution singing voice sample.BigVGAN recovers some harmonics with anti-aliasing filters, whereas Vocos does so without additional modules.
- Neural audio codec: Vocos notably outperforms EnCodec perceptually in DAPS reconstruction when both models are evaluated at matched 7.9M parameter counts.EnCodec artifacts are reflected more strongly in UTMOS than in PESQ and periodicity scores.
- Inference speed: Vocos runs approximately 13 times faster than HiFi-GAN and nearly 70 times faster than BigVGAN.The advantage is especially pronounced without GPU acceleration and is primarily attributed to ISTFT-based waveform synthesis instead of transposed convolutions.
5 CONCLUSIONS
Vocos bridges time-domain and Fourier-based neural vocoding by reconstructing Fourier coefficients while addressing phase wrapping. It matches state-of-the-art audio quality and improves computational efficiency through inverse fast Fourier transform upsampling.
- Vocos bridges time-domain and Fourier-based neural vocoding by accurately reconstructing Fourier time-frequency coefficients.Its generator is designed to handle phase wrapping during direct complex-valued spectrogram reconstruction.
- Vocos matches state-of-the-art audio quality while mitigating periodicity issues commonly observed in time-domain GANs.
- Inverse fast Fourier transform upsampling gives Vocos a significant computational efficiency advantage over traditional time-domain methods.
- The work advances neural vocoding by incorporating Fourier-based time-frequency representations.The source code and model weights are open-sourced for further exploration and application.
A MODIFIED DISCRETE COSINE TRANSFORM (MDCT)
MDCT provides a critically sampled time-frequency representation by producing fewer output coefficients from overlapping input blocks. Its relationship with the DFT enables fast FFT-based implementation.
- MDCT enables a critically sampled analysis/synthesis filter bank for high-quality audio coding.Its Time-Domain Aliasing Cancellation property allows perfect reconstruction of overlapping source-signal segments.
- The MDCT is defined over k = 0, 1, . . . , N −1, with N denoting the window length.
- MDCT produces N output coefficients from 2N input samples, allowing 50% block overlap without increasing the data rate.
- MDCT is related to the DFT through the Shifted Discrete Fourier Transform, enabling a fast FFT-based implementation.
A.1 VOCOS AND MDCT
MDCT offers compact, lower-dimensional audio representations, but its real-valued coefficients make generative modeling more challenging. Vocos explores symlog and signed parameterizations to produce MDCT coefficients.
- MDCT’s compact representation may reduce dimensionality and the number of data points required during generation.
- Naively predicting unnormalized MDCT coefficients led to slower convergence in preliminary experiments.
- Symlog compresses large coefficient magnitudes symmetrically around zero while preserving their signs.Its inverse is called symexp, and the pair corresponds to a symmetric logarithmic transformation.
- An alternative parameterization predicts MDCT coefficient magnitude and sign, using cosine activation to provide a periodic inductive bias.The final coefficients are represented as MDCT = exp(m) · cos(p).
- The MDCT variant is evaluated against the ISTFT baseline using objective metrics.
A.2 RESULTS
The MDCT-based Vocos variants underperform the STFT-based counterpart, indicating that STFT redundancy benefits GAN-based generative modeling. Accurate, frame-consistent coefficients remain necessary for MDCT alias cancellation.
- Both symexp and sign MDCT variants demonstrate significantly weaker performance than the STFT-based counterpart.
- STFT redundancy appears beneficial for generative tasks and aligns with findings that overcomplete Fourier bases improve training stability.
- MDCT’s lapped windows incorporate surrounding-frame aliases, requiring accurate and consistent coefficient predictions for TDAC.
A.3 FORWARD MDCT ALGORITHM
The forward MDCT algorithm uses FFT-based computation to transform windowed, overlapping audio frames into MDCT coefficients.
- The algorithm is realized with FFT-based computation.
- The procedure takes an audio signal x with frame length N and outputs MDCT coefficients X.
- Each frame uses an overlap of N/2 before applying a window function.
- After processing the frames, the procedure returns X.