Source-linked AI summary
EffVOC: Low-Delay Efficient Speech Waveform Reconstruction from Spectral Representations Without Phase
Renzheng Shi, Simon Welker, Timo Gerkmann, Tim Fingscheidt
TL;DR
Existing speech waveform reconstruction methods either require high delay, lose quality at low delay, or are limited to particular representations and bandwidths. EffVOC adapts an efficient low-delay vocoder to support amplitude-spectrum and Mel inputs for wideband and fullband synthesis. Its 20 ms models achieve top subjective MOS results, including 4.11 for the smallest fullband Mel model and scores within 0.05 MOS points of ground truth.
Problem
Existing methods face trade-offs among algorithmic delay, speech quality, complexity, and support for different input representations and bandwidths.
Method
EffVOC adapts a causal low-delay vocoder with efficient upsampling and recurrent layers for amplitude-spectrum or Mel inputs and wideband or fullband synthesis.
Results
20 ms EffVOC models achieve top subjective quality, with wideband and fullband systems ranking ahead of baselines and the smallest fullband Mel model reaching MOS 4.11.
Takeaways & Limitations
EffVOC provides a small, low-complexity, real-time-capable framework that establishes new subjective MOS results for wideband speech and fullband Mel synthesis.
Takeaways & Limitations
Algorithmic delay is defined by frame length, while converter delay and platform-dependent processing time are additional factors reported separately.
Abstract
from arXiv · showhide
The Griffin-Lim algorithm has been a seminal contribution for phase reconstruction from amplitude spectrograms, however, requiring (infinitely) high algorithmic delay. Its low-delay variant suffers in speech quality. Recent (generative) neural network methods improve on speech quality still at medium to high algorithmic delay, but often they are complex and optimized only for one specific input representation. We build upon an efficient low-delay speech vocoder and propose EffVOC, which supports synthesis of wideband (WB) or fullband (FB) speech from either amplitude spectrum or Mel coefficient inputs. We evaluate both input representations across multiple model sizes in a unified framework and compare against state of the art. Results show that our proposed low-delay (20 ms vs. 32 ms or more) efficient approach marks a new SOTA by achieving top-ranked subjective MOS scores (WB: 4.17/4.15, FB: 4.14/4.11) for amplitude spectrum/Mel representations, very close to ground truth.
1. Introduction
Existing phase-reconstruction methods trade waveform quality, flexibility, complexity, or delay. EffVOC addresses this gap with a unified low-delay model supporting wideband or fullband synthesis from amplitude spectra or Mel coefficients.
- Griffin-Lim reconstructs phase iteratively but typically needs many iterations, causing high computational cost and algorithmic delay.
- Neural phase-reconstruction methods can improve speed or quality, but hybrid approaches may depend heavily on traditional methods and direct predictors often require complex training.
- Utterance-based neural reconstruction has inherently high delay, while causal extensions can degrade performance and motivate specialized low-delay approaches.
- The prior low-delay vocoder achieves 20 ms delay but supports only Mel inputs and wideband synthesis, leaving amplitude-spectrum reconstruction and higher bandwidths open.
- EffVOC supports wideband or fullband synthesis from amplitude spectrum or Mel coefficient inputs within one reconstruction model.
2. Proposed Model
EffVOC is a causal, low-delay speech waveform reconstruction model that progressively upsamples input features while modeling temporal dependencies. It supports amplitude spectrogram or Mel inputs and wideband or fullband synthesis.
- Model design: EffVOC combines efficient upsampling with recurrent layers to model local feature dynamics and long-range temporal coherence.The architecture progressively increases temporal resolution toward the time-domain waveform while retaining recurrent modeling.
- Supported representations and bandwidths: EffVOC accepts either Mel coefficients or amplitude spectrograms and synthesizes either wideband or fullband speech.The architecture is mostly shared across bandwidths, with the final transposed-convolution stages adjusted for time-frequency resolution.
- Model design: The model uses two convolutional layers, two LSTM layers, and four transposed convolutional layers followed by residual blocks.Weight normalization is applied to convolutional layers, and causal convolutions are used throughout.
- Input representations: The amplitude-spectrum input retains the non-redundant frequency bins after applying a DFT to Hann-windowed speech frames.Mel inputs are obtained by applying a Mel filter bank and logarithmic scaling to the spectrogram representation.
- Delay definition: Algorithmic delay is determined by the processing frame length, while converter delay and platform processing time are reported separately.The paper uses real-time factor to report platform-dependent processing time.
3. Experimental Setup
The experiments evaluate EffVOC on the VCTK corpus using fixed wideband and fullband framing, varied model configurations, and instrumental and subjective measures. Comparisons include baselines using different delays, representations, and computational settings.
- Dataset: EffVOC is evaluated on the multi-speaker English-language VCTK dataset using a fixed test split shared with BAPEN.The corpus contains approximately 35.8 hours from 93 speakers, with a 2.5-hour validation set and wideband recordings resampled to 16 kHz.
- Signal processing: The experiments use 20 ms Hann windows with 5 ms frame shifts, corresponding to 320/80 samples for wideband and 960/240 for fullband.The corresponding DFT sizes are 512 and 1024, respectively.
- Model configurations: Wideband experiments vary model width across F ∈ {64, 32, 16, 8}, while fullband experiments fix F = 32 and vary stride configurations.The fullband setups use three different transposed-convolution stride patterns to examine model depth.
- Baselines: Baselines include methods evaluated with 32 ms windows and 16 ms shifts, alongside results taken directly from prior work.This setup enables comparison with MelFlow, DiffPhase, and BAPEN under their reported configurations.
- Evaluation: Evaluation combines intrusive quality, intelligibility, spectral, non-intrusive quality, and phonetic similarity metrics.The listed measures include PESQ-WB, POLQA, ESTOI, MCD, LSD, and NISQA.
- Subjective evaluation: Subjective MOS is computed over 160 files, with each file rated by eight listeners who passed an ITU-T P.808 check.The subset was selected to preserve the rank order and distribution of instrumental metrics relative to the full test set.
4. Results and Discussion
EffVOC provides low-delay waveform reconstruction across amplitude-spectrum and Mel inputs, with strong quality and efficiency in wideband and fullband speech. Its 20 ms models outperform or closely match higher-delay baselines across subjective quality, intelligibility, and fidelity measures.
- Wideband results: 4.31 vs. 4.26 PESQ-WB is achieved by EffVOC amplitude input versus BAPEN with 32 ms delay, using only 20 ms algorithmic delay.At similar model size, EffVOC reaches comparable PESQ-WB with substantially lower computational demand.
- Wideband results: Two-orders-of-magnitude lower GFLOPS are required by EffVOC F = 32 than MelFlow with 5 iterations, while achieving roughly the same speech quality and better ESTOI and signal fidelity.EffVOC also uses lower algorithmic delay and smaller model size, with slightly better real-time factors.
- Representation and model size: Amplitude and Mel representations perform comparably for F = 64 and F = 32, whereas amplitude inputs provide significantly better quality for smaller networks.Across Mel-input comparisons, EffVOC maintains stronger intelligibility and signal fidelity than competing methods.
- Fullband results: Amplitude-input models lead overall in fullband POLQA, intelligibility, and signal fidelity, while BAPEN remains strong in PESQ-WB but uses infinite delay.EffVOC also exceeds GLA on fullband POLQA.
5. Conclusions
EffVOC unifies low-delay speech waveform reconstruction from amplitude-spectrum and Mel inputs in a compact framework. Its 20 ms models achieve state-of-the-art subjective quality for wideband and fullband speech, including a 4.11 MOS score from the smallest fullband Mel model.
- EffVOC investigates amplitude-spectrum and Mel coefficient inputs for low-delay speech waveform reconstruction within one unified speech vocoder framework.
- 20 ms low-delay EffVOC models are small, low-complexity, and real-time capable while achieving new state of the art in wideband subjective MOS.
- 4.11 MOS is reached by the smallest proposed fullband EffVOC model with Mel input, outperforming all baselines and staying less than 0.05 MOS points below ground truth.