Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks

• Magnitude conversion removes STFT phase, while non-convex intensity constraints with unknown phase make spectrogram inversion challenging.The introduction notes that no known algorithm guarantees a globally optimal solution at low computational complexity.Source: §I · p. 1
• Existing inversion methods include iterative Griffin-Lim, deterministic SPSI, analytical Gaussian-window methods, and convex relaxation, but generic fixed objectives are inflexible for speech.Griffin-Lim can be slow because its operations are sequential; convex relaxation increases dimensionality.Source: §I · p. 1
• Sample-rate autoregressive speech models require inference about 16k–24k times per second, motivating the question of whether high-quality synthesis needs explicit autoregression.Some cited models instead autoregress at spectrogram-frame rates without a noticeable decrease in audio quality.Source: §I · p. 1
• MCNN uses non-autoregressive modeling and reports high audio quality with more than 300x real-time synthesis, with potential for integration into end-to-end audio models.Quality is quantified using human ratings, spectral convergence, and speaker classification accuracy.Source: §I · p. 1

• MCNN has parallel heads with the same layer types but different weights and initialization, learning to cooperate as an ensemble.Each head contains transposed-convolution layers, each applying a 1-D temporal convolution followed by an exponential linear unit.Source: §II · p. 2
• The product of layer strides and the spectrogram frame count is set to equal the waveform sample count, while filter widths control the local neighborhood used for upsampling.The input is an STFT-magnitude spectrogram; its hop and window parameters are assumed known.Source: §II · pp. 1–2
• Shared convolutional filters allow arbitrary-duration spectrogram input; scaled head outputs are summed and passed through a trainable scaled-softsign nonlinearity to bound the waveform.The number of filters is gradually reduced to one to produce the time-domain waveform.Source: §II · p. 2

• Spectral convergence emphasizes large spectral components, which helps especially in early training.The paper lists it first among the losses by observed empirical significance.Source: §III · p. 2
• Log-scale STFT-magnitude loss targets small-amplitude components, which tend to matter more in later training.The paper combines this term with spectral convergence and other losses.Source: §III · p. 2
• Instantaneous frequency loss uses a smoother phase-dependent metric because raw spectral phase is highly unstructured and difficult to fit.Source: §III · p. 2
• Weighted phase loss scales phase agreement by the magnitudes of both STFTs, focusing on high-amplitude components and improving numerical stability.The loss is minimized when the estimated and ground-truth STFT phases match.Source: §III · p. 2

A. Experimental setup

• Training uses preprocessed LibriSpeech: 960 hours of public-domain audiobooks from 2,484 speakers, sampled at 16 kHz.The dataset was designed for speech recognition and has lower audio quality than speech-synthesis datasets.Source: A. Experimental setup · p. 3
• The setup uses a 256-sample hop, 1,024-sample Hanning window, 2,048-point FFT, and an eight-layer transposed-convolution model.Each layer uses stride two, with filter width 13 and channels decreasing from 28 across layers.Source: A. Experimental setup · p. 3
• The model is trained for about 600k iterations with Adam and compared against Griffin-Lim and SPSI, with and without extra Griffin-Lim iterations.Training uses batch size 16 distributed across four GPUs with synchronous updates.Source: A. Experimental setup · p. 3

B. Synthesized audio waveform quality

• MCNN fits complicated waveform patterns with small phase errors between relevant high-amplitude spectral components.Fig. 2 compares the ground-truth waveform and spectrogram (left) with MCNN-generated outputs (right), including a zoomed portion.Source: B. Synthesized audio waveform quality · p. 3, Figure 2 · p. 3
• Human ratings (MOS) favor MCNN over Griffin-Lim, even with many iterations and SPSI initialization.SC-only MCNN is on par with Griffin-Lim, but some samples have background noise and less clear high-frequency harmonics; Fig. 3 contrasts SC-only training (top) with all losses (bottom).Source: B. Synthesized audio waveform quality · p. 3, Figure 3 · p. 3
• SC-only training can yield slightly better test-sample spectral convergence, but some outputs have background noise and less clear high-frequency harmonics.The paper reports that integrating other losses further improves audio quality.Source: B. Synthesized audio waveform quality · p. 3

C. Generalization and optimization to a particular speaker

• MCNN trained on LibriSpeech maintains audio quality on an unseen speaker, while a speaker-specific model yields a very small quality gap with ground truth.The speaker-specific model is trained on that speaker’s audio with reoptimized hyperparameters.Source: C. Generalization and optimization to… · p. 3

D. Representation learning of the frequency basis

• For constant-frequency inputs, MCNN synthesizes sinusoids; inputs with multiple frequency bands yield superposed sinusoids with long-window phase coherence.The synthesized representations become poorer at frequencies beyond human speech, attributed to increased train-test mismatch; Fig. 4 depicts the waveform outputs for the stated sinusoid inputs.Source: D. Representation learning of the… · p. 4, Figure 4 · p. 4

E. Deployment considerations

• ∼5.2M samples/sec gives MCNN ∼330 times faster-than-real-time synthesis on a Nvidia Tesla P100 GPU.Griffin-Lim runtime is ∼20 times slower for 50 iterations and ∼60 times slower for 150 iterations.Source: E. Deployment considerations · p. 4
• MCNN’s compute intensity is 61 FLOPs/byte, versus 1.9 FLOPs/byte for Griffin-Lim, while its computational complexity is ∼2.2 GFLOPs/sec.The paper reports that MCNN’s complexity is slightly higher than that of 150 Griffin-Lim iterations.Source: E. Deployment considerations · p. 4
• MCNN’s shorter critical path and lower DRAM bandwidth requirement support parallelization and utilization on modern multi-core processors.The paper attributes the much shorter runtime in part to these architectural properties.Source: E. Deployment considerations · p. 4

V. CONCLUSIONS

• MCNN provides very fast waveform synthesis without noticeably sacrificing perceptual quality, and generalizes to unseen speech or speakers.It is trained on a large-scale speech dataset.Source: §V · p. 4
• Larger audio datasets are expected to close the quality gap with ground truth, and future work could integrate MCNN into end-to-end generative audio systems.The conclusion names text-to-speech and audio style transfer as examples.Source: §V · p. 4
• The comparison uses TensorFlow operations without specific kernel optimizations and a GPU Griffin-Lim implementation using TensorFlow FFT and inverse FFT operations.Hardware-specific kernel optimization could yield further improvements.Source: §V · p. 4

A. Complexity modeling

• The complexity model counts point-wise operations as 1 FLOP, matrix multiplication as 2mnp FLOPs, and a real FFT of length N as 2.5N log2(N) FLOPs.It excludes hardware-specific logic-level costs and register memory moves, and notes that FLOPs do not represent some power costs such as data movement.Source: A. Complexity modeling · p. 5

B. Analysis of contributions of multiple heads

• Different heads focus on different waveform portions and frequency bands, although their individual outputs have very low intelligibility.Head 2 mostly focuses on low-frequency components; heads share an architecture, and their random initial weights determine their later focus.Source: B. Analysis of contributions of… · p. 5
• The combined head outputs can yield highly natural-sounding waveforms, including frequencies absent from individual outputs after nonlinear combination.Fig. 5 shows the synthesized waveform and log-STFT above the eight constituent-head outputs; waveforms are normalized per head and small-amplitude STFT components are thresholded.Source: B. Analysis of contributions of… · p. 5, Figure 5 · p. 6